Premiere version : mise en route du suivi.
[auf_roundup.git] / doc / overview.txt
2Roundup: an Issue-Tracking System for Knowledge Workers
5:Authors: Ka-Ping Yee (original_), Richard Jones (implementation)
7.. _original: original_overview.html
9.. contents::
15Roundup is an issue-tracking system called which will manage a
16number of issues (with properties such as "description", "priority",
17and so on) and provides the ability to:
19(a) submit new issues,
20(b) find and edit existing issues, and
21(c) discuss issues with other participants.
23Roundup facilitates communication among the participants by managing
24discussions and notifying interested parties when issues are edited.
30A typical software project requires the management of
31many tasks, usually distributed among several collaborators.
32In fact, any project team
33could use a tool for sorting out and discussing all the
34relevant issues. A common approach is to set up some kind
35of "to-do" list that people can share.
37However, to address the overall problem we need much more
38than just a shared to-do list; we need to
39manage a growing body of knowledge and experience to help a
40team collaborate effectively on a project. The issue-tracking
41tool becomes a nexus for communication: the Grand Central
42Station of the group intelligence.
44The primary focus of this design is to help
45developers work together well, not to provide a customer
46service interface to the developers. This is not to say that
47the design is to be made unsuitable for customers to use.
48Rather, it is assumed that many of the same qualities
49that are good for supporting development (see below)
50are also good for non-developers using the system.
51Additional niceties
52for providing a safe or simplified interface to clients are
53intentionally deferred for later consideration.
55A good issue-tracking system should have at least the
56following properties:
58**Low barrier to participation**
59 The usefulness of the tool depends entirely on the
60 information people contribute to it. It must be made
61 as easy as possible to submit new issues and contribute
62 information about existing issues.
64**Straightforward navigation**
65 It should be easy for users to extract information they need
66 from the system to direct their decisions and tasks.
67 They should be able to get a decent overview of
68 things as well as finding specific information when
69 they know what they're after.
71**Controlled information flow**
72 The users must have control over how much information and
73 what information they get. A common flaw of some issue-tracking
74 systems is that they inundate users with so much useless
75 e-mail that people avoid the system altogether.
77With a nod to the time-honoured computer science tradition
78of "filling in the fourth quadrant", we note that
79there are really four kinds of information flow
80going on here. The three mentioned qualities
81really address the first three quadrants of this 2-by-2 matrix,
841. User push: a user submits information to the system.
852. User pull: a user queries for information from the system.
863. System push: the system sends information out to users.
874. System pull: the system solicits information from users.
89An example of the fourth kind of flow is voting.
90Voting isn't described in this design,
91but it should be noted as a potential enhancement.
94Guiding Principles
98 It is a strong requirement
99 that the tool be accessible and understandable. It should
100 be fairly obvious what different parts of the interface do,
101 and the inner mechanisms should operate in ways that most
102 users can easily predict.
105 We aim to optimize for minimum effort to do the most common
106 operations, and best use of resources like screen real estate
107 to maximize the amount of information that we summarize and present.
110 We try to avoid making
111 unnecessary assumptions that would restrict the applicability
112 of the tool. For example, there is no reason why one might
113 not also want to use this tool to manage a design process,
114 non-software projects, or organizational decisions.
116**Persistence** We
117 prefer hiding or reclassifying information to deleting it.
118 This helps support the collection of statistics later.
119 If records are never destroyed, there is little danger
120 in providing access to a larger community, and logging yields
121 accountability, which may encourage better behaviour.
124Data Model
127Roundup stores a number of *items*, each of
128which can have several properties and an associated discussion.
129The properties can be used to classify or search for items.
130The discussion is a sequence of e-mail messages.
131Each item is identified by a unique number, and has
132an activity log which
133records the time and content of edits made on its properties.
134The log stays fairly small since the design intentionally
135provides only small data types as item properties, and
136encourages anything large to be attached to
137e-mail where it becomes part of the discussion.
138The next section explains how items are organized.
141The Hyperdatabase
144Often when classifying information we are
145asked to select exactly one of a number of categories
146or to fit it into a rigid hierarchy. Yet things
147only sometimes fall into one category; often,
148a piece of information may be related to several concepts.
150For example, forcing each item into a single keyword
151category is not just suboptimal but counterproductive:
152seekers of that
153item may expect to find it in a different category
154and conclude that the item is not present in the
155database -- which has them *worse* off
156than if the items were not categorized at all.
158Some systems try to alleviate this problem by
159allowing items to appear at multiple locations
160in a tree, as with "aliases" or "symbolic links" in
161a filesystem, for example. This does help somewhat,
162but we want to be even more flexible
163by allowing the
164organization of items into sets that may freely
165intersect. Rather than putting each item at exactly
166one place in an overall "grand scheme", a item can
167belong to as many sets as are appropriate.
169If we choose to represent the sets themselves as items
170and set membership as a link between items,
171we're now ready to present the definition of a
174A *hyperdatabase* is a collection of *items*
175that may be hyperlinked to
176each other (hence the name "hyperdatabase").
177Each item carries a collection of key-value pairs,
178where some of the values may be links to other items.
179Any item may have an arbitrary number of outgoing and
180incoming links. Hyperdatabases are able to efficiently
181answer queries such as "what items link to this item?"
182and "what items does this item link to?"
187There are several reasons for building our
188own kind of database for Roundup rather than using an existing one.
190Requiring the installation of a full-blown third-party
191SQL database system would probably deter many potential
192users from attempting to set up Roundup;
193yet a real relational database would be too
194complicated to implement on our own.
196On the other hand, a hyperdatabase can be implemented fairly easily
197using one of the Python DBM modules, so we can
198take the "batteries-included" approach and provide it
199as part of the system. It's easier to build and understand
200than a true relational database (in accordance with our guiding
201principle of *simplicity*), but provides
202most of the query functionality we want.
204A hyperdatabase is well suited for finding the intersection
205of a number of sets in which items belong. We expect that
206most of the queries people want to do will be of this
207form, rather than complicated SQL queries. For example, a
208typical request might be
209"show me all critical items related to security".
210The ability to store arbitrary key-value pairs and links
211on items gives it more flexibility than an RDBMS.
213Users are not going to be making thousands of queries
214per second, so it makes sense to optimize for simplicity
215and flexibility rather than performance.
217.. img: images/hyperdb.png
220Roundup's Hyperdatabase
223For our application, we store each item as a item in a
224hyperdatabase. The item's properties are stored
225as key-value pairs on its item.
226Several types of properties are allowed:
227*string*, *number*, *boolean*, *date*, *interval, *link*,
228and *multlink*. Another type, *password*, is a special type
229of string and it's only used internally to Roundup.
231The *string* type is for short, free-form strings.
232String properties are not intended to contain large
233amounts of text, and it is recommended that they be presented
234as one-line fields to encourage brevity. A *number* is a special
235type of string that represents a numeric value. A *boolean* is
236further constrained to be a *true* or *false* value.
238The *date* type is for calendar dates and times. An *interval*
239is the time between two dates.
241The *link* type denotes a single selection from a number of
242options. A *link* property entails a link from the item
243possessing the property to the item representing the chosen option.
245The *multilink* type is for a list of links to any
246number of other items in the in the database. A *multilink*
247property, for example, can be used to refer to related items
248or keyword categories relevant to an item.
250For Roundup, all items have four properties that are not customizable:
2521. a *date* property named **creation**
2532. a *link* property named **creator**
2543. a *date* property named **activity**
256These properties represent the date of the creation of the item, who
257created it, and when the last change was made.
259Further, all *issue* items have an additional four properties:
2611. a *string* property named **title**
2622. a *multilink* property named **nosy**
2633. a *multilink* property named **messages**
2644. a *multilink* property named **files**
2655. a *multilink* property named **superseder**
267The **title** property is a short one-line description of the item.
268The detailed description can go in the first e-mail message of the
269item's messages spool.
271The **nosy** property contains a list of
272the people who are interested in an item. This
273mechanism is explained in the section on `Submission and Discussion`_.
275Each message added to the item goes in the **messages** spool - any
276attached files go in the **files** spool.
278The **superseder** property is used to
279support the splitting, joining, or replacing of items.
280When several items need to be
281joined into a single item, all the old items
282link to the new item in their **superseder**
284When an item needs to be split apart, the item
285references all the new items in its **superseder**
287We can easily list all active items just by checking
288for an empty **superseder** property, and trace
289the path of an item's origins by querying the hyperdatabase
290for links.
292Users of the system are also represented by items in the
293hyperdatabase, containing properties
294like the user's e-mail address, login name, and password.
296The Default Schema
299It is hoped that the hyperdatabase together with the
300specializations mentioned above for Roundup will be
301applicable in a variety of situations
302(in accordance with our guiding principle of *generality*).
304To address the problem at hand, we need
305a specific schema for items applied particularly to software development.
306Again, we are trying to keep the schema simple: too many
307options make it tougher for someone to make a good choice::
309 # IssueClass automatically gets these properties:
310 # title = String()
311 # messages = Multilink("msg")
312 # files = Multilink("file")
313 # nosy = Multilink("user")
314 # superseder = Multilink("issue")
315 # (it also gets the Class properties creation, activity and creator)
316 issue = IssueClass(db, "issue",
317 assignedto=Link("user"), keyword=Multilink("keyword"),
318 priority=Link("priority"), status=Link("status"))
320The **assignedto** property assigns
321responsibility for an item to a person or a list of people.
322The **keyword** property places the
323item in an arbitrary number of relevant keyword sets (see
324the section on `Browsing and Searching`_).
326The **prority** and **status** values are initially:
328=========== =====================================
329Priority Description
330=========== =====================================
331"critical" panic: work is stopped!
332"urgent" important, but not deadly
333"bug" lost work or incorrect results
334"feature" want missing functionality
335"wish" avoidable bugs, missing conveniences
336=========== =====================================
338============= =====================================
339Status Description
340============= =====================================
341"unread" submitted but no action yet
342"deferred" intentionally set aside
343"chatting" under review or seeking clarification
344"need-eg" need a reproducible example of a bug
345"in-progress" understood; development in progress
346"testing" we think it's done; others, please test
347"done-cbb" okay for now, but could be better
348"resolved" fix has been released
349============= =====================================
351As previously mentioned, each item gets an activity log.
352Whenever a property on an item is changed, the log
353records the time of the change, the user making the change,
354and the old and new values of the property. This permits
355the later gathering of statistics (for example, the average time
356from submission to resolution).
358We do not specify or enforce a state transition graph,
359since making the system rigid in that fashion is probably more
360trouble than it's worth.
361Experience has shown that there are probably
362two convenient automatic state transitions:
3641. from **unread** to **chatting** when e-mail is written about an item
3652. from **testing** to **resolved** when a new release of the software is made
367Beyond these, in accordance with our principle of *generality*,
368we allow access to the hyperdatabase
369API so that scripts can automate transitions themselves or
370be triggered by changes in the database.
373User Interface
376Roundup provides its services through two main interfaces:
377e-mail and the Web.
378This division is chosen to optimize the most common tasks.
380E-mail is best suited for
381the submission of new items since most people are most comfortable
382with composing long messages in their own favourite e-mail client.
383E-mail also permits them to mention URLs or attach files relevant
384to their submission. Indeed, in many cases people are already
385used to making requests by sending e-mail to a mailing list
386of people; they can do exactly the same thing to use Roundup
387without even thinking about it.
388Similarly, people are already
389familiar with holding discussions in e-mail, and plenty of
390valuable usage conventions and software tools already exist for that medium.
392The Web, on the other hand, is best suited for summarizing
393and seeking information, because it can present an interactive
394overview of items. Since the Web has forms, it's also
395the best place to edit items.
398Submission and Discussion
401The system needs an address for receiving mail
402and an address that forwards mail to all participants.
403Each item has its own list
404of interested parties, known as its *nosy list*.
405Here's how nosy lists work:
4071. New items are always submitted by sending an e-mail message
408 to Roundup. The "Subject:" field becomes the description
409 of the new item.
410 The message is saved in the mail spool of the new item,
411 and copied to the list of all participants
412 so everyone knows that a new item has been added.
413 The new item's nosy list initially contains the submitter.
4152. All e-mail messages sent by Roundup have their "Reply-To:"
416 field set to Roundup's address, and have the item's
417 number in the "Subject:" field. Thus, any replies to the
418 initial announcement and subsequent threads are all received
419 by Roundup. Roundup notes the item number in the "Subject:"
420 field of each incoming message and appends the message
421 to the appropriate spool.
4233. Any incoming e-mail tagged with an item number is copied
424 to all the people on the item's nosy list,
425 and any users found in the "From:", "To:", or "Cc:" fields
426 are automatically added to the nosy list. Whenever a user
427 edits an item's properties in the Web interface, they are
428 also added to the nosy list.
430The effect is like each item having its own little mailing list,
431except that no one ever has to worry about subscribing to
432anything. Indicating interest in an issue is sufficient, and if you
433want to bring someone new into the conversation, all you need to do
434is "Cc:" a message to them. It turns out that no one ever has to worry
435about unsubscribing, either: the nosy lists are so specific in scope
436that the conversation tends to die down by itself when the issue is
437resolved or people no longer find it sufficiently important.
439Each nosy list is like an asynchronous chat room,
440lasting only a short time (typically five or ten messages)
441and involving a small group of people. However, that
442group is the *right* group of people:
443only those who express interest in an item in some way
444ever end up on the list, so no one gets spammed with mail they
445don't care about, and no one who *wants*
446to see mail about a particular item needs to be left
447out, for they can easily join in, and just as easily
448look at the mail spool on an item to catch up on any
449messages they might have missed.
451We can take this a step further and
452permit users to monitor particular keywords or classifications of items
453by allowing other kinds of items to also have their own nosy lists.
454For example, a manager could be on the
455nosy list of the priority value item for "critical", or a
456developer could be on the nosy list of the keyword value item for "security".
457The recipients are then determined by the union of the nosy lists on the
458item and all the items it links to.
460Using many small, specific mailing lists results
461in much more effective communication than one big list.
462Taking away the effort of subscribing and unsubscribing
463gives these lists the "feel" of being cheap and
466The transparent capture of the mail spool attached to each
467issue also yields a nice knowledge repository over time.
473Since Roundup is intended to support arbitrary user-defined
474schema for item properties, the editing interface must be
475automatically generated from the schema. The configuration for
476Roundup will include a template describing how to lay out the
477properties to present a UI for inspecting and editing items.
478For example::
480 <tr>
481 <th class="required">Priority</th>
482 <td tal:content="structure context/priority/menu">priority</td>
483 <th>Status</th>
484 <td tal:content="structure context/status/menu">status</td>
485 </tr>
487To display the editing form for an item, Roundup inserts
488an HTML form widget where it encounters an expression like
489``tal:content="structure context/priority/menu"``.
490Each type has its own appropriate editing widget:
492- *string* and *number* properties appear as text fields
493- *boolean* properties appear as a yes/no selection
494- *date* and *interval* properties appear as text fields
495- *link* properties appear as selection lists
496- *multilink* properties appear as multiple-selection lists
497 or text fields with pop-up widgets for larger selections.
499We foresee the use of custom date fields for things like deadlines,
500so input fields for *date* properties support a
501simple way of specifying relative dates (such as "3w" for
502"three weeks from now").
504The **superseder** property is a special case:
505although it is more efficient to store a **superseder**
506property in the superseded item, it makes more sense to provide
507a "supersedes" edit field on the superseding item. We use
508a special widget on items for this purpose (a text field containing
509a comma-separated list of items). Links in the **superseder** property
510appear on both the superseding and superseded items to
511facilitate navigating an item's pedigree.
513After the editing widgets, the item inspection page shows
514a "note" text box and then a display of the messages in the
515discussion spool. This field
516lets you enter a note explaining your change when you edit the
517item, and the note is included in the notification message that
518goes out to tell the interested parties on the nosy list of
519your edits.
521Browsing and Searching
524The ideal we would like to achieve is to make searching as
525much like browsing as possible: the user simply clicks about
526on things that seem interesting, and the information narrows
527down comfortably until the goal is in sight. This is preferable
528to trying to digest a screen filled with widgets and buttons
529or entering a search expression in some arcane algebraic syntax.
531While a one-shot search may be appropriate when you're
532looking for a single item and you know exactly what you want, it's
533not very helpful when you want an overview of
534things ("Gee, there are a lot more high-priority items than
535there were last week!") or trying to do comparisons ("I have
536some time today, so who is busiest and could most use some help?")
538The browsing interface presents filtering
539functionality for each of the properties in the schema. As with
540editing, the interface is generated from a template
541describing how to lay out the properties.
542Each type of property has its own appropriate filtering widget:
544- *string* properties appear as text fields supporting
545 case-insensitive substring match
546- *date* properties appear as a text field which accepts a date
547 range with start, end or both
548- *link* properties appear as a group of selectable options
549 (the filter selects the *union* of the sets of items
550 associated with the active options)
551- *multilink* properties appear as a group of selectable options
552 (the filter selects the *intersection* of the sets of items
553 associated with the active options)
555For a *multilink* property like **keyword**,
556one possibility is to show, as hyperlinks, the keywords whose
557sets have non-empty intersections with the currently displayed set of
558items. Sorting the keywords by popularity seems
559reasonable. Clicking on a keyword then narrows both the list of items
560and the list of keywords. This gives some of the feel of walking
561around a directory tree -- but without the restriction of having
562to select keywords in a particular hierarchical order, and without
563the need to travel all the way to the leaves of the tree before
564any items are visible.
566Below the filtering form is a listing of items, with their
567properties displayed in a table. Rows in the table are
568generated from a template, as with the editing interface.
569This listing is the central overview of the system, and it
570should aim to maximize the density of
571useful information in accordance with our guiding principle of
572*efficiency*. Colour may be used to indicate
573the status of each item to help the eye sift through the index quickly.
575Roundup sorts items
576in groups by priority, and then within groups by the date
577of last activity. This reveals at a glance where discussion is
578most active, and provides an easy way for anyone to move an issue
579up in the list.
581The page produced by a given set of browsing options constitutes
582an *index*. The options should all be part of the query
583parameters in the URL so that views may be bookmarked. An index
586- search strings for string properties
587- date ranges for date properties
588- acceptable values for choice properties
589- required values for reference properties
590- a sorting key
591- a grouping key
592- a list of properties for which to display filtering widgets
594Our default index is:
596- all **status** values except "resolved"
597- show **priority** and **fixer**
598- grouping by **priority** in sections
599- sorting by decreasing **activity** date
601The starting URL for Roundup immediately presents the listing of
602items generated by this default index, with no preceding query screen.