.. _New-Style-Build-Steps:

New-Style Build Steps
=====================

In Buildbot-0.9.0, many operations performed by BuildStep subclasses return a Deferred.
As a result, custom build steps which call these methods will need to be rewritten.

Buildbot-0.8.9 supports old-style steps natively, while new-style steps are emulated.
Buildbot-0.9.0 supports new-style steps natively, while old-style steps are emulated.
Later versions of Buildbot wil not support old-style steps at all.
All custom steps should be rewritten in the new style as soon as possible.

Buildbot distinguishes new-style from old-style steps by the presence of a :py:meth:`~buildbot.process.buildstep.BuildStep.run` method.
If this method is present, then the step is a new-style step.

Summary of Changes
++++++++++++++++++

 * New-style steps have a ``run`` method that is simpler to implement than the old ``start`` method.
 * Many methods are now asynchronous (return Deferreds), as they perform operations on the database.
 * Logs are now implemented by a completely different class.
   This class supports the same log-writing methods (``addStderr`` and so on), although they are now asynchronous.
   However, it does not support log-reading methods such as ``getText``.
   It was never advisable to handle logs as enormous strings.
   New-style steps should, instead, use a LogObserver or (in Buildbot-0.9.0) fetch log lines bit by bit using the data API.
 * :py:class:`buildbot.process.buildstep.LoggingBuildStep` is deprecated and cannot be uesd in new-style steps.
   Mix in :py:class:`buildbot.process.buildstep.ShellMixin` instead.

Rewriting ``start``
+++++++++++++++++++

If your custom buildstep implements the ``start`` method, then rename that method to ``run`` and set it up to return a Deferred, either explicitly or via ``inlineCallbacks``.
The value of the Deferred should be the result of the step (one of the codes in :py:mod:`buildbot.status.results`), or a Twisted failure instance to complete the step as EXCEPTION.
The new ``run`` method should *not* call ``self.finished`` or ``self.failed``, instead signalling the same via Deferred.

For example, the following old-style ``start`` method ::


    def start(self):  ## old style
        cmd = remotecommand.RemoteCommand('stat', {'file': self.file })
        d = self.runCommand(cmd)
        d.addCallback(lambda res: self.convertResult(cmd))
        d.addErrback(self.failed)

Becomes ::

    @defer.inlineCallbacks
    def run(self):  ## new style
        cmd = remotecommand.RemoteCommand('stat', {'file': self.file })
        yield self.runCommand(cmd)
        yield self.convertResult(cmd)

Newly Asynchronous Methods
++++++++++++++++++++++++++

The following methods now return a Deferred:

 * :py:meth:`buildbot.process.buildstep.BuildStep.addLog`
 * ``log.addStdout``
 * ``log.addStderr``
 * ``log.addHeader``
 * ``log.finish`` (see "Log Objects", below)
 * :py:meth:`buildbot.process.remotecommand.RemoteCommand.addStdout`
 * :py:meth:`buildbot.process.remotecommand.RemoteCommand.addStderr`
 * :py:meth:`buildbot.process.remotecommand.RemoteCommand.addHeader`
 * :py:meth:`buildbot.process.remotecommand.RemoteCommand.addToLog`
 * :py:meth:`buildbot.process.buildstep.BuildStep.addCompleteLog`
 * :py:meth:`buildbot.process.buildstep.BuildStep.addHTMLLog`
 * :py:meth:`buildbot.process.buildstep.BuildStep.addURL`

Any custom code in a new-style step that calls these methods must handle the resulting Deferred.
In some cases, that means that the calling method's signature will change.
For example ::

    def summarize(self):  ## old-style
        for m in self.MESSAGES:
            if counts[m]:
                self.addCompleteLog(m, "".join(summaries[m]))
            self.setProperty("count-%s" % m, counts[m], "counter")

Is a synchronous function, not returning a Deferred.
However, when converted to a new-style test, it must handle Deferreds from the methods it calls, so it must be asynchronous.
Syntactically, ``inlineCallbacks`` makes the change fairly simple::

    @defer.inlineCallbacks
    def summarize(self):  ## new-style
        for m in self.MESSAGES:
            if counts[m]:
                yield self.addCompleteLog(m, "".join(summaries[m]))
            self.setProperty("count-%s" % m, counts[m], "counter")

However, this method's callers must now handle the Deferred that it returns.
All methods that can be overridden in custom steps can return a Deferred.

Properties
++++++++++

Good news!
The API for properties is the same synchronous API as was available in old-style steps.
Properties are handled synchronously during the build, and persisted to the database at completion of each step.

Log Objects
+++++++++++

Old steps had two ways of interacting with logfiles, both of which have changed.

The first is writing to logs while a step is executing.
When using :py:meth:`~buildbot.process.buildstep.BuildStep.addCompleteLog` or :py:meth:`~buildbot.process.buildstep.BuildStep.addHTMLLog`, this is straightforward, except that in new-style steps these methods return a Deferred.

The second method is via :py:meth:`buildbot.process.buildstep.BuildStep.addLog`.
In new-style steps, the returned object (via Deferred) has the following methods to add log content:

 * :py:meth:`~buildbot.process.log.StreamLog.addStdout`
 * :py:meth:`~buildbot.process.log.StreamLog.addStderr`
 * :py:meth:`~buildbot.process.log.StreamLog.addHeader`
 * :py:meth:`~buildbot.process.log.Log.finish`

All of these methods now return Deferreds.
None of the old log-reading methods are available on this object:

 * ``hasContents``
 * ``getText``
 * ``readLines``
 * ``getTextWithHeaders``
 * ``getChunks``

If your step uses such methods, consider using a :class:`~buildbot.process.logobserver.LogObserver` instead, or using the Data API to get the required data.

The undocumented and unused ``subscribeConsumer`` method of logfiles has also been removed.

The :py:meth:`~buildbot.process.log.Log.subscribe` method now takes a callable, rather than an instance, and does not support catchup.
This method was primarily used by :py:class:`~buildbot.process.logobserver.LogObserver`, the implementation of which has been modified accordingly.
Any other uses of the subscribe method should be refactored to use a :py:class:`~buildbot.process.logobserver.LogObserver`.

Status Strings
++++++++++++++

The ``self.step_status.setText`` and ``setText2`` methods have been removed.
Similarly, the ``_describe`` and ``describe`` methods are not used in new-style steps.
In fact, steps no longer set their status directly.

Instead, steps call :py:meth:`buildbot.process.buildstep.BuildStep.updateSummary` whenever the status may have changed.
This method calls :py:meth:`~buildbot.process.buildstep.BuildStep.getCurrentSummary` or :py:meth:`~buildbot.process.buildstep.BuildStep.getResultSummary` as appropriate and update displays of the step's status.
Steps override the latter two methods to provide appropriate summaries.

Statistics
++++++++++

Support for statistics has been moved to the ``BuildStep`` and ``Build`` objects.
Calls to ``self.step_status.setStatistic`` should be rewritten as ``self.setStatistic``.