<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="revision" content="$Id: mailbox-format.html,v 1.9 2010/01/06 17:01:29 murch Exp $" />
<meta name="author" content="Bron Gondwana" />
<title>Mailbox API</title>
</head>
<body>
<h1>Mailbox API</h1>
<h2>Intro</h2>
<p>The Mailbox API is implemented in <tt>imap/mailbox.h</tt> and
<tt>imap/mailbox.c</tt>. It wraps the data structures of the
<tt>cyrus.header</tt>, <tt>cyrus.index</tt> and <tt>cyrus.cache</tt>
files in a psuedo-object-oriented way, allowing easy changes to
the mailbox while keeping the internal cached data structures
consistent.</p>
<h2>Opening and closing</h2>
<pre>
struct mailbox *mailbox = NULL;
int r;
const char *mboxname = "user.brong";
r = mailbox_open_iwl(mboxname, &mailbox);
// or
r = mailbox_open_irl(mboxname, &mailbox);
// or
r = mailbox_open_exlusive(mboxname, &mailbox);
if (r) return r;
do_stuff(mailbox);
mailbox_close(&mailbox);
</pre>
<p>It is always necessary to obtain an index lock when opening
a mailbox, because the index header read must be consistent.
The locks are as follows:</p>
<table border="1">
<tr>
<th>Function</th><th>Namelock</th><th>Index Lock</th>
</tr>
<tr>
<td>mailbox_open_iwl</td><td>Shared</td><td>Exclusive</td>
</tr>
<tr>
<td>mailbox_open_irl</td><td>Shared</td><td>Shared</td>
</tr>
<tr>
<td>mailbox_open_exclusive</td><td>Exclusive</td><td>Exclusive</td>
</tr>
</table>
<p>It should never be necessary to call <tt>mailbox_open_exclusive</tt>,
but it's included for completeness. Use <tt>mailbox_open_iwl</tt> if
you expect to need to write to the index (or even if you're not sure)
and <tt>mailbox_open_irl</tt> when you know you're only reading from
the file and wish to allow other readers to work concurrently.</p>
<p>Many actions are delayed until the mailbox is closed, or even until
the <em>last</em> mailbox is closed for things that require an
exclusive namelock to perform like deletion or repack. See below
under "delayed actions" for more detail.</p>
<p>To avoid opening the same file multiple times, the mailbox API
refcounts open mailboxes. If you open the same mailbox again (i.e.
a URL fetch or status command on the currently select mailbox) then
the same mailbox will be returned. It must be unlocked (see below
or the open command will return IMAP_MAILBOX_LOCKED). The matching
close will reduce the refcount, and only the final close will do
the cleanup actions.</p>
<h2>Locking and unlocking</h2>
<p>You can keep a mailbox "open", maintaining the namelock, while
releasing the index lock to allow other processes to make changes
to the mailbox. By holding the namelock, you know that record
numbers won't change, and the underlying message files won't be
deleted.</p>
<p><tt>mailbox_close</tt> will call <tt>mailbox_unlock_index</tt>
if the index is still locked, so it is not neccessary to explicitly
unlock the index before closing.</p>
<pre>
r = mailbox_unlock_index(mailbox, NULL);
// sleep on user input...
r = mailbox_lock_index(mailbox, LOCK_SHARED);
// or
r = mailbox_lock_index(mailbox, LOCK_EXCLUSIVE);
</pre>
<p>For example, <tt>mailbox_unlock_index</tt> and
<tt>mailbox_lock_index</tt> are used extensively
by the index module, allowing an imap client to
maintain a long lived connection selected to a
mailbox and know that messages won't magically
disappear from under it - yet at the same time
allow new mail delivery to happen or other imap
connections to query the mailbox.</p>
<p>If you have built an accurate statuscache item for
the locked mailbox, you can pass this as the second
parameter to mailbox_index_unlock. If there have been
any changes, mailbox_index_unlock will invalidated the
statuscache. If you give it the new value, then it will
store that value instead. For example:</p>
<pre>
struct statusdata sdata;
index_status(state, &sdata);
/* RECENT is zero for everyone else because we wrote a new
* recentuid! */
sdata.recent = 0;
mailbox_unlock_index(state->mailbox, &sdata);
</pre>
<p>See "delayed actions" below for delayed actions performed
during an unlock.</p>
<h2>Creating, renaming and deleting</h2>
<p><b>WARNING:</b> These functions only change the mailbox
files on disk. They don't update the mailboxes.db records
or contact murder servers. In most cases you are probably
looking for the <tt>mboxlist_</tt> functions instead.</p>
<p>Creating a mailbox is somewhat longwinded - as there are many
optional parameters.</p>
<pre>
int mailbox_create(const char *name, const char *part, const char *acl,
const char *uniqueid, int options, unsigned uidvalidity,
struct mailbox **mailboxptr);
</pre>
<p>Most interesting to note is that on success, <tt>mailboxptr</tt> will
contain the same mailbox that <tt>mailbox_open_exclusive</tt> above would
have returned, with an exclusive namelock and an exclusive index lock.
This allows you to perform other consistency operations after creating
the mailbox with a full guarantee that no other process will even be
able to know of the mailbox's existence! You can still roll-back by
deleting the mailbox and the next process will get the namelock and see
no mailbox with that name.</p>
<pre>
int mailbox_rename_copy(struct mailbox *oldmailbox,
const char *newname, const char *newpart,
const char *userid, int ignorequota,
struct mailbox **newmailboxptr);
</pre>
<p>Very similar to mailbox_create - the new mailbox is created with
an exclusive name lock and returned. The old mailbox must be passed
in with an <b>exclusive index lock</b> but is fine with a shared
namelock, as it will be passed to <tt>mailbox_delete</tt>.</p>
<pre>
int mailbox_delete(struct mailbox **mailboxptr);
</pre>
<p>Just like <tt>mailbox_close</tt> above, this closes the mailbox.
Before it does so, it sets the OPT_MAILBOX_DELETED option flag in
the index header. The interesting work is actualy done in
<tt>mailbox_close</tt>. See below under "delayed actions".</p>
<p><tt>mailbox_delete</tt> requires an exclusive index lock, but
can complete quite happily with only a shared namelock.
<h2>Reading and writing records</h2>
<p>Ok - so you have a mailbox, it's opened, and the index is
locked. Time to start reading and writing some records!</p>
<p>At the mailbox level there is no concept of "message numbers"
from imap, only "record numbers". The canonical variable name
to refer to record numbers is <tt>recno</tt>. All records are
read and written using <tt>struct index_record</tt> values.</p>
<p>Here at the API definitions used for reading and writing:</p>
<pre>
int mailbox_read_index_record(struct mailbox *mailbox,
uint32_t recno,
struct index_record *record);
int mailbox_rewrite_index_record(struct mailbox *mailbox,
struct index_record *record);
int mailbox_append_index_record(struct mailbox *mailbox,
struct index_record *record);
int mailbox_commit(mailbox);
</pre>
<p>An example of iterating through a mailbox</p>
<pre>
uint32_t recno;
struct index_record record;
int make_changes;
for (recno = 1; recno <= mailbox->i.num_records; recno++) {
if (mailbox_read_index_record(mailbox, recno, &record))
fatal("invalid record", EC_SOFTWARE); // or return an error
if (record.system_flags & FLAG_EXPUNGED)
continue; // skip expunged records
make_changes = do_stuff(mailbox, &record);
if (make_changes)
mailbox_rewrite_index_record(mailbox, &record);
}
</pre>
<p>NOTE: <tt>mailbox_rewrite_index_record</tt> doesn't need a recno,
as that's cached inside the index_record struct.</p>
<p>NOTE: You need an exclusively locked index to use rewrite or append,
but only a shared index lock to use read.</p>
<p>There are a range of consistency checks done to ensure that a
rewrite doesn't violate IMAP semantics (an expunged message can
never be unexpunged, UIDs can't change, etc) and the internal
tracking counts and quota data are updated as well. They will
be committed at unlock time, see "delayed actions"</p>
<p>If you don't set the <tt>record.silent<tt> field to a true value
before rewriting or appending, the <tt>record.modseq</tt> and
<tt>record.last_updated</tt> values will be changed. This allows
condstore to work correctly.</p>
<h3>Appending</h3>
<p>To append a record, the file must have already been copied into
place (XXX - plan to move to a stage based system where the mailbox
API handles the staging, but that's not finished yet) and been parsed
into the record struct. The UID must be set already, and must be greater
than the UID of any existing record in the mailbox. There are a range
of consistency checks done.</p>
<p>The internal consistency counts are updated by append as well.</p>
<h3>Committing</h3>
<p>When you have finished making any changes, you need to "commit".
This will write the updated values for any index header fields,
rewite the <tt>cyrus.header</tt> file if needed and fsync all
changes to disk.</p>
<p>It is a fatal error to unlock (or close) a mailbox that has had
changes without committing, as it can leave the mailbox in a
corrupted state.</p>
<h3>Cache records</h3>
Cache records are accessed through <tt>record.crec</tt> which is not
filled by read_index_record. The cache file is only read and mapped
into memory as needed, so you if you want to access cache records,
the basic API is as follows:
<pre>
int mailbox_cacherecord(struct mailbox *mailbox,
struct index_record *record);
const char *cacheitem_base(struct index_record *record, int field);
unsigned cacheitem_size(struct index_record *record, int field);
struct buf *cacheitem_buf(struct index_record *record, int field);
</pre>
<p>You must always call <tt>mailbox_cacherecord</tt> on a record
before trying to access any of the cache items. "<tt>field</tt>"
above is the individual field (there are 10) in the cache record.
There's more information on those fields in the mailbox internal
format documentation.</p>
<pre>
for (recno = 1; recno <= mailbox->i.num_records; recno++) {
if (mailbox_read_index_record(mailbox, recno, &record))
fatal("invalid record", EC_SOFTWARE); // or return an error
if (record.system_flags & FLAG_EXPUNGED)
continue; // skip expunged records
if (mailbox_cacherecord(mailbox, &record))
fatal("failed to read cache", EC_SOFTWARE);
...
envelope_length = cacheitem_size(&record, CACHE_ENVELOPE);
}
</pre>
<p>See <tt>imap/mailbox.h</tt> for the full list of constants.</p>
<h2>Delayed Actions</h2>
<p>Here's the bit you've been waiting for! What happens during
unlock and close</p>
<h3>first, unlock</h3>
<p>Anything that makes any changes sets the mailbox->has_changed
flag. If this is set, then before the index gets unlocked:</p>
<ul>
<li>the updatenotifier (idle) is called</li>
<li><tt>sync_log_mailbox</tt> (replication) gets called</li>
<li>the statuscache value gets erased (or replaced if you passed
in an updated value).</li>
</ul>
<h3>then: close</h3>
<p>First close looks to see if the first_expunged date is old
(based on the configuration variable <tt>expunge_days</tt>),
if so - and there's an exclusive index lock - then it will
attempt an expunge cleanup. This allows for "automatic"
cleanup rather than running cyr_expire if wanted.</p>
<p>next the index is unlocked (see above)</p>
<p>third, any "unlink" commands scheduled for email files are
run. These can't be done until after the mailbox_commit to
ensure consistency - the file isn't deleted until the record
is written as unlinked! But we save the unlink until now so
that other tasks aren't waiting for the index lock while the
unlinks run. Unlink is expensive in IO and time.</p>
<p>finally we check for MAILBOX_NEEDS_REPACK or MAILBOX_DELETED
option flags. If either is sets, then we make a non-blocking
attempt to get an exclusive namelock. If the non-blocking
attempt fails, then another process has the mailbox open, so
save the cleanup for them! If it succeeds, then go ahead
with either <tt>mailbox_delete_cleanup</tt> or
<tt>mailbox_index_repack</tt> as appropriate.</p>
<p>After this it's just a matter of releasing malloc'd memory
and finally releasing the name lock.</p>
</body>
</html>
