pushlog: expose obsolescence metadata (bug 1286426); r=glandium
authorGregory Szorc <gps@mozilla.com>
Thu, 14 Jul 2016 11:54:21 -0700
changeset 8969 6900f22f81a21153f82389478368b19a0d86de64
parent 8968 cc3360b720c65f901fa67070f57bdd6311415f0a
child 8970 b7d763081bf2873710a9aaa73c28483603c5dc28
child 8972 3367357851c96e6ff33e14aa1f1c89a6d04c44cc
push id1034
push userbmo:gps@mozilla.com
push dateTue, 19 Jul 2016 18:10:24 +0000
reviewersglandium
bugs1286426
pushlog: expose obsolescence metadata (bug 1286426); r=glandium The next step towards handling obsolescence data in pushlog is to expose it to consumers so they can do intelligent things. This commit changes behavior in a number of ways. First, "successors" and "precursors" nodes lists are added to changeset entries if the repository has obsolescence data. Second, a "obsoletechangesets" key is now present on each push entry. It operates like "changesets" except it only exposes changesets that are obsolete. I decided to create a new container for obsolete changesets because adding obsolete changesets to the existing container is too risky: existing consumers of the pushlog wouldn't know to e.g. look for a key to identify a changeset as obsolete. By introducing a new container, existing consumers have to explicitly opt in to looking at obsolete changesets. And, since key additions are backwards compatible, we don't need to introduce a v3 of the API to expose this new data! There should be enough metadata in the pushlog now for consumers to determine whether a changeset was rewritten and what the previous or new changeset is. We may also want to update Pulse notifications to include similar metadata so Pulse consumers don't need to consult the pushlog. MozReview-Commit-ID: 1H1Kwo7KSxc
docs/hgmo/pushlog.rst
hgext/pushlog-legacy/pushlog-feed.py
hgext/pushlog-legacy/tests/test-obsolescence.t
--- a/docs/hgmo/pushlog.rst
+++ b/docs/hgmo/pushlog.rst
@@ -190,16 +190,19 @@ push. e.g.::
        "91826025c77c6a8e5711735adaa9766dd4eac7fc",
        "25f2a69ac7ac2919ef35c0b937b862fbb9e7e1f7"
       ],
       "date": 1227196396,
       "user": "gszorc@mozilla.com"
      }
    }
 
+An optional ``obsoletechangesets`` key may also be present in each push.
+Read below for more.
+
 Version 2
 ^^^^^^^^^
 
 Version 2 introduces a container for pushes so that additional metadata
 can be communicated in the main object in the payload. Here is an
 example payload::
 
    {
@@ -246,34 +249,44 @@ changesets
    push. If ``full`` is specified, entries are objects containing
    changeset metadata (see below).
 
    Changesets are in DAG/revlog order with the tip-most changeset last.
 
    The array may be empty. This can occur if changesets from this push
    are now hidden/obsolete.
 
+obsoletechangesets
+   (optional) An array of 40 character changeset SHA-1s of now obsolete
+   changesets included in the push.
+
+   The DAG order relationship between ``changesets`` and ``obsoletechangesets``
+   is strictly speaking undefined.
+
+   This key is only present if the repository has obsolescence data and the
+   push has changesets that are now obsolete.
+
 date
    Integer seconds since UNIX epoch that the push occurred.
 
    For pushes that take a very long time (more than a single second),
    the data will be recorded towards the end of the push, just before
    the transaction is committed to Mercurial. Although, this is an
    implementation details.
 
    There is no guarantee of strict ordering between dates. i.e. the
    ``date`` of push ID ``N + 1`` could be less than the ``date`` of push
    ID ``N``. Such is how clocks work.
 
 user
    The string username that performed the push.
 
-If ``full`` is specified, each entry in the ``changesets`` array will be
-an object instead of a string. Each object will have the following
-properties:
+If ``full`` is specified, each entry in the ``changesets`` and
+``obsoletechangesets`` array will be an object instead of a string.
+Each object will have the following properties:
 
 node
    The 40 byte hex SHA-1 of the changeset.
 
 parents
    An array of 1 or 2 elements containing the 40 byte hex SHA-1 of the
    parent changesets. Merges have 2 entries. Root changesets have the
    value ``0000000000000000000000000000000000000000``.
@@ -290,29 +303,45 @@ branch
    ``default`` is the default branch in Mercurial.
 
 tags
    An array of string tags belonging to this changeset.
 
 files
    An array of filenames that were changed by this changeset.
 
+precursors
+   (optional) An array of 40 character hex SHA-1 nodes identifying
+   *precursor* nodes.
+
+   *Precursor* nodes are essentially previously versions of this changeset.
+
+   Precursor nodes come from obsolescence data. This key won't exist if
+   there are no precursor nodes for this changeset.
+
+   The precursor changesets are hidden and not available to normal Mercurial
+   operations. However, querying the pushlog for their info *may* return
+   results.
+
 Here's an example::
 
    {
      "author": "Eugen Sawin <esawin@mozilla.com>",
      "branch": "default",
      "desc": "Bug 1110212 - Strong randomness for Android DNS resolver. r=sworkman",
      "files": [
        "other-licenses/android/res_init.c"
      ],
      "node": "ee4fe2ec168e719e822dabcdd797c0cff9ce2407",
      "parents": [
        "803bc910c45a875d9d76dc689c45dd91a1e02e23"
      ],
+     "precursors": [
+       "d313a202a85e114000f669c2fcb49ad42376ac04"
+     ],
      "tags": []
    }
 
 Writing Agents that Consume Pushlog Data
 ========================================
 
 It is common to want to write tools or services that consume pushlog
 data. For example, you may wish to perform processing of new commits as
--- a/hgext/pushlog-legacy/pushlog-feed.py
+++ b/hgext/pushlog-legacy/pushlog-feed.py
@@ -13,16 +13,17 @@ from mercurial.hgweb.common import (
     ErrorResponse,
     HTTP_OK,
     paritygen,
 )
 from mercurial.node import hex, nullid
 from mercurial import (
     demandimport,
     error,
+    obsolete,
     templatefilters,
 )
 
 sys.path.append(os.path.dirname(__file__))
 
 with demandimport.deactivated():
     from parsedatetime import parsedatetime as pdt
 
@@ -475,48 +476,63 @@ def pushlogHTML(web, req, tmpl):
                 startdate='startdate' in req.form and req.form['startdate'][0] or '1 week ago',
                 enddate='enddate' in req.form and req.form['enddate'][0] or 'now',
                 querydescription=query.description(),
                 archives=web.archivelist("tip"))
 
 def pushes_worker(query, repo, full):
     """Given a PushlogQuery, return a data structure mapping push IDs
     to a map of data about the push."""
+    haveobs = bool(repo.obsstore)
     pushes = {}
     for id, user, date, node in query.entries:
         id = str(id)
 
         # Create the pushes entry first. It is OK to have empty
         # pushes if nodes from the pushlog no longer exist.
         if id not in pushes:
             pushes[id] = {
                 'user': user,
                 'date': date,
                 'changesets': [],
             }
 
         try:
             ctx = repo[node]
+            nodekey = 'changesets'
         # Changeset is hidden
         except error.FilteredRepoLookupError:
-            continue
+            # Try to find the hidden changeset so its metadata can be used.
+            try:
+                ctx = repo.unfiltered()[node]
+            except error.LookupError:
+                continue
+
+            nodekey = 'obsoletechangesets'
 
         if full:
             node = {
                 'node': ctx.hex(),
                 'author': ctx.user(),
                 'desc': ctx.description(),
                 'branch': ctx.branch(),
                 'parents': [c.hex() for c in ctx.parents()],
                 'tags': ctx.tags(),
                 'files': ctx.files()
             }
 
+            # Only expose obsolescence metadata if the repo has some.
+            if haveobs:
+                precursors = obsolete.precursormarkers(ctx)
+                precursors = [hex(m.precnode()) for m in precursors]
+                if precursors:
+                    node['precursors'] = precursors
+
         # we get the pushes in reverse order
-        pushes[id]['changesets'].insert(0, node)
+        pushes[id].setdefault(nodekey, []).insert(0, node)
 
     return {'pushes': pushes, 'lastpushid': query.lastpushid}
 
 def pushes(web, req, tmpl):
     """WebCommand to return a data structure containing pushes."""
     query = pushlogSetup(web.repo, req)
     data = pushes_worker(query, web.repo, 'full' in req.form)
 
--- a/hgext/pushlog-legacy/tests/test-obsolescence.t
+++ b/hgext/pushlog-legacy/tests/test-obsolescence.t
@@ -60,31 +60,35 @@
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 0 changes to 2 files
   recorded push in pushlog
   2 new obsolescence markers
 
-FIXME Hidden changesets should not be exposed to version 1
+Hidden changesets exposed as list under obsoletechangesets in version 1
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?version=1"
   200
   {
       "1": {
           "changesets": [
               "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
           ],
           "date": \d+, (re)
           "user": "user@example.com"
       },
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+              "d313a202a85e114000f669c2fcb49ad42376ac04"
+          ],
           "user": "user@example.com"
       },
       "3": {
           "changesets": [
               "b3641753ee63b166fad7c5f10060b0cbbc8a86b0",
               "62eebb2f0f00195f9d965f718090c678c4fa414d"
           ],
           "date": \d+, (re)
@@ -95,16 +99,18 @@ FIXME Hidden changesets should not be ex
               "418a63f508062fb2eb9130065c5ddc7908dd5949",
               "d129109168f0ed985e51b0f86df256acdcfcfe45"
           ],
           "date": \d+, (re)
           "user": "user@example.com"
       }
   }
 
+obsolete changeset metadata exposed under full with version 1
+
   $ httpjson "http://localhost:$HGPORT/json-pushes?version=1&full=1"
   200
   {
       "1": {
           "changesets": [
               {
                   "author": "test",
                   "branch": "default",
@@ -120,16 +126,44 @@ FIXME Hidden changesets should not be ex
               }
           ],
           "date": \d+, (re)
           "user": "user@example.com"
       },
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file0",
+                  "files": [
+                      "file0"
+                  ],
+                  "node": "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+                  "parents": [
+                      "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
+                  ],
+                  "tags": []
+              },
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file1",
+                  "files": [
+                      "file1"
+                  ],
+                  "node": "d313a202a85e114000f669c2fcb49ad42376ac04",
+                  "parents": [
+                      "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                  ],
+                  "tags": []
+              }
+          ],
           "user": "user@example.com"
       },
       "3": {
           "changesets": [
               {
                   "author": "test",
                   "branch": "default",
                   "desc": "file2",
@@ -167,56 +201,66 @@ FIXME Hidden changesets should not be ex
                   "desc": "file0",
                   "files": [
                       "file0"
                   ],
                   "node": "418a63f508062fb2eb9130065c5ddc7908dd5949",
                   "parents": [
                       "62eebb2f0f00195f9d965f718090c678c4fa414d"
                   ],
+                  "precursors": [
+                      "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                  ],
                   "tags": []
               },
               {
                   "author": "test",
                   "branch": "default",
                   "desc": "file1",
                   "files": [
                       "file1"
                   ],
                   "node": "d129109168f0ed985e51b0f86df256acdcfcfe45",
                   "parents": [
                       "418a63f508062fb2eb9130065c5ddc7908dd5949"
                   ],
+                  "precursors": [
+                      "d313a202a85e114000f669c2fcb49ad42376ac04"
+                  ],
                   "tags": [
                       "tip"
                   ]
               }
           ],
           "date": \d+, (re)
           "user": "user@example.com"
       }
   }
 
-FIXME Hidden changesets should not be exposed to version 2
+Hidden changesets exposed as list with version 2
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?version=2"
   200
   {
       "lastpushid": 4,
       "pushes": {
           "1": {
               "changesets": [
                   "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
               ],
               "date": \d+, (re)
               "user": "user@example.com"
           },
           "2": {
               "changesets": [],
               "date": \d+, (re)
+              "obsoletechangesets": [
+                  "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+                  "d313a202a85e114000f669c2fcb49ad42376ac04"
+              ],
               "user": "user@example.com"
           },
           "3": {
               "changesets": [
                   "b3641753ee63b166fad7c5f10060b0cbbc8a86b0",
                   "62eebb2f0f00195f9d965f718090c678c4fa414d"
               ],
               "date": \d+, (re)
@@ -228,16 +272,18 @@ FIXME Hidden changesets should not be ex
                   "d129109168f0ed985e51b0f86df256acdcfcfe45"
               ],
               "date": \d+, (re)
               "user": "user@example.com"
           }
       }
   }
 
+Hidden changeset metadata exposed under version 2 with full
+
   $ httpjson "http://localhost:$HGPORT/json-pushes?version=2&full=1"
   200
   {
       "lastpushid": 4,
       "pushes": {
           "1": {
               "changesets": [
                   {
@@ -255,16 +301,44 @@ FIXME Hidden changesets should not be ex
                   }
               ],
               "date": \d+, (re)
               "user": "user@example.com"
           },
           "2": {
               "changesets": [],
               "date": \d+, (re)
+              "obsoletechangesets": [
+                  {
+                      "author": "test",
+                      "branch": "default",
+                      "desc": "file0",
+                      "files": [
+                          "file0"
+                      ],
+                      "node": "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+                      "parents": [
+                          "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
+                      ],
+                      "tags": []
+                  },
+                  {
+                      "author": "test",
+                      "branch": "default",
+                      "desc": "file1",
+                      "files": [
+                          "file1"
+                      ],
+                      "node": "d313a202a85e114000f669c2fcb49ad42376ac04",
+                      "parents": [
+                          "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                      ],
+                      "tags": []
+                  }
+              ],
               "user": "user@example.com"
           },
           "3": {
               "changesets": [
                   {
                       "author": "test",
                       "branch": "default",
                       "desc": "file2",
@@ -302,41 +376,47 @@ FIXME Hidden changesets should not be ex
                       "desc": "file0",
                       "files": [
                           "file0"
                       ],
                       "node": "418a63f508062fb2eb9130065c5ddc7908dd5949",
                       "parents": [
                           "62eebb2f0f00195f9d965f718090c678c4fa414d"
                       ],
+                      "precursors": [
+                          "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                      ],
                       "tags": []
                   },
                   {
                       "author": "test",
                       "branch": "default",
                       "desc": "file1",
                       "files": [
                           "file1"
                       ],
                       "node": "d129109168f0ed985e51b0f86df256acdcfcfe45",
                       "parents": [
                           "418a63f508062fb2eb9130065c5ddc7908dd5949"
                       ],
+                      "precursors": [
+                          "d313a202a85e114000f669c2fcb49ad42376ac04"
+                      ],
                       "tags": [
                           "tip"
                       ]
                   }
               ],
               "date": \d+, (re)
               "user": "user@example.com"
           }
       }
   }
 
-FIXME Hidden changesets handled properly on feed
+Hidden changesets dropped in feed
 
   $ http --no-headers "http://localhost:$HGPORT/atom-pushlog"
   200
   
   <?xml version="1.0" encoding="ascii"?>
   <feed xmlns="http://www.w3.org/2005/Atom">
    <id>http://*:$HGPORT/pushlog</id> (glob)
    <link rel="self" href="http://*:$HGPORT/pushlog"/> (glob)
@@ -511,29 +591,35 @@ Specifying a fromchange with a hidden ch
                   "desc": "file0",
                   "files": [
                       "file0"
                   ],
                   "node": "418a63f508062fb2eb9130065c5ddc7908dd5949",
                   "parents": [
                       "62eebb2f0f00195f9d965f718090c678c4fa414d"
                   ],
+                  "precursors": [
+                      "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                  ],
                   "tags": []
               },
               {
                   "author": "test",
                   "branch": "default",
                   "desc": "file1",
                   "files": [
                       "file1"
                   ],
                   "node": "d129109168f0ed985e51b0f86df256acdcfcfe45",
                   "parents": [
                       "418a63f508062fb2eb9130065c5ddc7908dd5949"
                   ],
+                  "precursors": [
+                      "d313a202a85e114000f669c2fcb49ad42376ac04"
+                  ],
                   "tags": [
                       "tip"
                   ]
               }
           ],
           "date": \d+, (re)
           "user": "user@example.com"
       }
@@ -542,43 +628,107 @@ Specifying a fromchange with a hidden ch
 Specifying a tochange with a hidden changeset works
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?startID=1&tochange=ae13d9da6966"
   200
   {
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+              "d313a202a85e114000f669c2fcb49ad42376ac04"
+          ],
           "user": "user@example.com"
       }
   }
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?startID=1&tochange=ae13d9da6966&full=1"
   200
   {
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file0",
+                  "files": [
+                      "file0"
+                  ],
+                  "node": "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+                  "parents": [
+                      "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
+                  ],
+                  "tags": []
+              },
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file1",
+                  "files": [
+                      "file1"
+                  ],
+                  "node": "d313a202a85e114000f669c2fcb49ad42376ac04",
+                  "parents": [
+                      "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                  ],
+                  "tags": []
+              }
+          ],
           "user": "user@example.com"
       }
   }
 
 Specifying a hidden changeset works
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?changeset=ae13d9da6966"
   200
   {
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+              "d313a202a85e114000f669c2fcb49ad42376ac04"
+          ],
           "user": "user@example.com"
       }
   }
 
   $ httpjson "http://localhost:$HGPORT/json-pushes?changeset=ae13d9da6966&full=1"
   200
   {
       "2": {
           "changesets": [],
           "date": \d+, (re)
+          "obsoletechangesets": [
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file0",
+                  "files": [
+                      "file0"
+                  ],
+                  "node": "ae13d9da6966307c98b60987fb4fedc2e2f29736",
+                  "parents": [
+                      "96ee1d7354c4ad7372047672c36a1f561e3a6a4c"
+                  ],
+                  "tags": []
+              },
+              {
+                  "author": "test",
+                  "branch": "default",
+                  "desc": "file1",
+                  "files": [
+                      "file1"
+                  ],
+                  "node": "d313a202a85e114000f669c2fcb49ad42376ac04",
+                  "parents": [
+                      "ae13d9da6966307c98b60987fb4fedc2e2f29736"
+                  ],
+                  "tags": []
+              }
+          ],
           "user": "user@example.com"
       }
   }