feat(pack): add initial lockfile tracking

Problem: Some use cases require or benefit from persistent on disk
  storage of plugin data (a.k.a. "lockfile"):

  1. Allow `update()` to act on not-yet-active plugins. Currently if
    `add()` is not yet called, then plugin's version is unknown
    and `update()` can't decide where to look for changes.

  2. Efficiently know plugin's dependencies without having to read
    'pkg.json' files on every load for every plugin. This is for the
    future, after there is `packspec` support (or other declaration of
    dependencies on plugin's side).

  3. Allow initial install to check out the exact latest "working" state
    for a reproducible setup. Currently it pulls the latest available
    `version.`

  4. Ensure that all declared plugins are installed, even if lazy loaded.
    So that later `add()` does not trigger auto-install (when there
    might be no Internet connection, for example) and there is no issues
    with knowing which plugins are used in the config (so even never
    loaded rare plugins are still installed and can be updated).

  5. Allow `add()` to detect if plugin's spec has changed between
    Nvim sessions and act accordingly. I.e. either set new `src` as
    origin or enforce `version.` This is not critical and can be done
    during `update()`, but it might be nice to have.

Solution: Add lockfile in JSON format that tracks (adds, updtes,
  removes) necessary data for described use cases. Here are the required
  data that enables each point:

    1. `name` -> `version` map.
    2. `name` -> `dependencies` map.
    3. `name` -> `rev` map. Probably also requires `name` -> `src` map
       to ensure that commit comes from correct origin.
    4. `name` -> `src` map. It would be good to also track the order,
       but that might be too many complications and redundant together
       with point 2.
    5. Map from `name` to all relevant spec fields. I.e. `name` -> `src`
       and `name` -> `version` for now. Storing data might be too much,
       but can be discussed, of course.

  This commit only adds lockfile tracking without implementing actual
  use cases. It is stored in user's config directory and is suggested to
  be tracked via version control.

  Example of a lockfile:

  ```json
  {
    # Extra nesting to more future proof.
    "plugins": {
      "plug-a": {
        "ref": "abcdef1"
        "src": "https://github.com/user/plug-a",
        # No `version` means it was `nil` (infer default branch later)
      },
      "plug-b": {
        "dependencies": ["plugin-a", "plug-c"],
        "src": "https://github.com/user/plug-b",
        "ref": "bcdefg2",
        # Enclose string `version` in quotes
        "version": "'dev'"
      },
      "plug-c": {
        "src": "https://github.com/user/plug-c",
        "ref": "cdefgh3",
        # Store `vim.version.Range` via its `tostring()` output
        "version": ">=0.0.0",
      }
    }
  }
  ```

runtime/doc/pack.txt

@@ -221,6 +221,12 @@ Uses Git to manage plugins and requires present `git` executable of at least
 version 2.36. Target plugins should be Git repositories with versions as named
 tags following semver convention `v<major>.<minor>.<patch>`.
 
+The latest state of all managed plugins is stored inside a *vim.pack-lockfile*
+located at `$XDG_CONFIG_HOME/nvim/nvim-pack-lock.json`. It is a JSON file that
+is used to persistently track data about plugins. For a more robust config
+treat lockfile like its part: put under version control, etc. Should not be
+edited by hand or deleted.
+
 Example workflows ~
 
 Basic install and management:
@@ -260,11 +266,13 @@ Basic install and management:
     show confirmation buffer in a separate tabpage.
   • Review changes. To confirm all updates execute |:write|. To discard
     updates execute |:quit|.
+  • (Optionally) |:restart| to start using code from updated plugins.
 
 Switch plugin's version:
 • Update 'init.lua' for plugin to have desired `version`. Let's say, plugin
   named 'plugin1' has changed to `vim.version.range('*')`.
-• |:restart|. The plugin's actual state on disk is not yet changed.
+• |:restart|. The plugin's actual state on disk is not yet changed. Only
+  plugin's `version` in |vim.pack-lockfile| is updated.
 • Execute `vim.pack.update({ 'plugin1' })`.
 • Review changes and either confirm or discard them. If discarded, revert any
   changes in 'init.lua' as well or you will be prompted again next time you
@@ -272,7 +280,7 @@ Switch plugin's version:
 
 Freeze plugin from being updated:
 • Update 'init.lua' for plugin to have `version` set to current revision. Get
-  it with `:=vim.pack.get({ 'plug-name' })[1].rev` (looks like `abc12345`).
+  it from |vim.pack-lockfile| (plugin's field `rev`; looks like `abc12345`).
 • |:restart|.
 
 Unfreeze plugin to start receiving updates:

runtime/lua/vim/pack.lua

@@ -14,6 +14,12 @@
 ---least version 2.36. Target plugins should be Git repositories with versions
 ---as named tags following semver convention `v<major>.<minor>.<patch>`.
 ---
+---The latest state of all managed plugins is stored inside a [vim.pack-lockfile]()
+---located at `$XDG_CONFIG_HOME/nvim/nvim-pack-lock.json`. It is a JSON file that
+---is used to persistently track data about plugins.
+---For a more robust config treat lockfile like its part: put under version control, etc.
+---Should not be edited by hand or deleted.
+---
 ---Example workflows ~
 ---
 ---Basic install and management:
@@ -57,11 +63,13 @@
 ---      show confirmation buffer in a separate tabpage.
 ---    - Review changes. To confirm all updates execute |:write|.
 ---      To discard updates execute |:quit|.
+---    - (Optionally) |:restart| to start using code from updated plugins.
 ---
 ---Switch plugin's version:
 ---- Update 'init.lua' for plugin to have desired `version`. Let's say, plugin
 ---named 'plugin1' has changed to `vim.version.range('*')`.
 ---- |:restart|. The plugin's actual state on disk is not yet changed.
+---  Only plugin's `version` in |vim.pack-lockfile| is updated.
 ---- Execute `vim.pack.update({ 'plugin1' })`.
 ---- Review changes and either confirm or discard them. If discarded, revert
 ---any changes in 'init.lua' as well or you will be prompted again next time
@@ -69,7 +77,7 @@
 ---
 ---Freeze plugin from being updated:
 ---- Update 'init.lua' for plugin to have `version` set to current revision.
----Get it with `:=vim.pack.get({ 'plug-name' })[1].rev` (looks like `abc12345`).
+---Get it from |vim.pack-lockfile| (plugin's field `rev`; looks like `abc12345`).
 ---- |:restart|.
 ---
 ---Unfreeze plugin to start receiving updates:
@@ -190,13 +198,72 @@ local function git_get_tags(cwd)
   return tags == '' and {} or vim.split(tags, '\n')
 end
 
--- Plugin operations ----------------------------------------------------------
+-- Lockfile -------------------------------------------------------------------
 
 --- @return string
 local function get_plug_dir()
   return vim.fs.joinpath(vim.fn.stdpath('data'), 'site', 'pack', 'core', 'opt')
 end
 
+--- @class (private) vim.pack.LockData
+--- @field rev string Latest recorded revision.
+--- @field src string Plugin source.
+--- @field version? string|vim.VersionRange Plugin `version`, as supplied in `spec`.
+
+--- @class (private) vim.pack.Lock
+--- @field plugins table<string, vim.pack.LockData> Map from plugin name to its lock data.
+
+--- @type vim.pack.Lock
+local plugin_lock
+
+local function lock_get_path()
+  return vim.fs.joinpath(vim.fn.stdpath('config'), 'nvim-pack-lock.json')
+end
+
+local function lock_read()
+  if plugin_lock then
+    return
+  end
+  local fd = uv.fs_open(lock_get_path(), 'r', 438)
+  if not fd then
+    plugin_lock = { plugins = {} }
+    return
+  end
+  local stat = assert(uv.fs_fstat(fd))
+  local data = assert(uv.fs_read(fd, stat.size, 0))
+  assert(uv.fs_close(fd))
+  plugin_lock = vim.json.decode(data) --- @type vim.pack.Lock
+
+  -- Deserialize `version`
+  for _, l_data in pairs(plugin_lock.plugins) do
+    local version = l_data.version
+    if type(version) == 'string' then
+      l_data.version = version:match("^'(.+)'$") or vim.version.range(version)
+    end
+  end
+end
+
+local function lock_write()
+  -- Serialize `version`
+  local lock = vim.deepcopy(plugin_lock)
+  for _, l_data in pairs(lock.plugins) do
+    local version = l_data.version
+    if version then
+      l_data.version = type(version) == 'string' and ("'%s'"):format(version) or tostring(version)
+    end
+  end
+
+  local path = lock_get_path()
+  vim.fn.mkdir(vim.fs.dirname(path), 'p')
+  local fd = assert(uv.fs_open(path, 'w', 438))
+
+  local data = vim.json.encode(lock, { indent = '  ', sort_keys = true })
+  assert(uv.fs_write(fd, data))
+  assert(uv.fs_close(fd))
+end
+
+-- Plugin operations ----------------------------------------------------------
+
 --- @param msg string|string[]
 --- @param level ('DEBUG'|'TRACE'|'INFO'|'WARN'|'ERROR')?
 local function notify(msg, level)
@@ -532,6 +599,8 @@ local function checkout(p, timestamp, skip_same_sha)
 
   git_cmd({ 'checkout', '--quiet', p.info.sha_target }, p.path)
 
+  plugin_lock.plugins[p.spec.name].rev = p.info.sha_target
+
   trigger_event(p, 'PackChanged', 'update')
 
   -- (Re)Generate help tags according to the current help files.
@@ -561,6 +630,8 @@ local function install_list(plug_list, confirm)
     git_clone(p.spec.src, p.path)
     p.info.installed = true
 
+    plugin_lock.plugins[p.spec.name].src = p.spec.src
+
     -- Do not skip checkout even if HEAD and target have same commit hash to
     -- have new repo in expected detached HEAD state and generated help files.
     checkout(p, timestamp, false)
@@ -698,17 +769,34 @@ function M.add(specs, opts)
   end
   plugs = normalize_plugs(plugs)
 
-  -- Install
+  -- Pre-process
-  --- @param p vim.pack.Plug
+  lock_read()
-  local plugs_to_install = vim.tbl_filter(function(p)
+  local plugs_to_install = {} --- @type vim.pack.Plug[]
-    return not p.info.installed
+  local needs_lock_write = false
-  end, plugs)
+  for _, p in ipairs(plugs) do
+    -- TODO(echasnovski): check that lock's `src` is the same as in spec.
+    -- If not - cleanly reclone (delete directory and mark as not installed).
+    local p_lock = plugin_lock.plugins[p.spec.name] or {}
+    needs_lock_write = needs_lock_write or p_lock.version ~= p.spec.version
+    p_lock.version = p.spec.version
+    plugin_lock.plugins[p.spec.name] = p_lock
 
+    if not p.info.installed then
+      plugs_to_install[#plugs_to_install + 1] = p
+      needs_lock_write = true
+    end
+  end
+
+  -- Install
   if #plugs_to_install > 0 then
     git_ensure_exec()
     install_list(plugs_to_install, opts.confirm)
   end
 
+  if needs_lock_write then
+    lock_write()
+  end
+
   -- Register and load those actually on disk while collecting errors
   -- Delay showing all errors to have "good" plugins added first
   local errors = {} --- @type string[]
@@ -899,6 +987,7 @@ function M.update(names, opts)
     return
   end
   git_ensure_exec()
+  lock_read()
 
   -- Perform update
   local timestamp = get_timestamp()
@@ -925,6 +1014,7 @@ function M.update(names, opts)
   run_list(plug_list, do_update, progress_title)
 
   if opts.force then
+    lock_write()
     feedback_log(plug_list)
     return
   end
@@ -950,6 +1040,7 @@ function M.update(names, opts)
     end
     run_list(plugs_to_checkout, do_checkout, 'Applying updates')
 
+    lock_write()
     feedback_log(plugs_to_checkout)
   end)
 end
@@ -967,6 +1058,8 @@ function M.del(names)
     return
   end
 
+  lock_read()
+
   for _, p in ipairs(plug_list) do
     trigger_event(p, 'PackChangedPre', 'delete')
 
@@ -974,8 +1067,12 @@ function M.del(names)
     active_plugins[p.path] = nil
     notify(("Removed plugin '%s'"):format(p.spec.name), 'INFO')
 
+    plugin_lock.plugins[p.spec.name] = nil
+
     trigger_event(p, 'PackChanged', 'delete')
   end
+
+  lock_write()
 end
 
 --- @inlinedoc

test/functional/plugin/pack_spec.lua

@@ -310,6 +310,14 @@ local function is_jit()
   return exec_lua('return package.loaded.jit ~= nil')
 end
 
+local function get_lock_path()
+  return vim.fs.joinpath(fn.stdpath('config'), 'nvim-pack-lock.json')
+end
+
+local function get_lock_tbl()
+  return vim.json.decode(fn.readblob(get_lock_path()))
+end
+
 -- Tests ======================================================================
 
 describe('vim.pack', function()
@@ -326,6 +334,7 @@ describe('vim.pack', function()
 
   after_each(function()
     vim.fs.rm(pack_get_dir(), { force = true, recursive = true })
+    vim.fs.rm(get_lock_path(), { force = true })
   end)
 
   teardown(function()
@@ -413,6 +422,80 @@ describe('vim.pack', function()
       eq('plugindirs main', exec_lua('return require("plugindirs")'))
     end)
 
+    it('creates lockfile', function()
+      local helptags_rev = git_get_hash('HEAD', 'helptags')
+      exec_lua(function()
+        vim.pack.add({
+          { src = repos_src.basic, version = 'some-tag' },
+          { src = repos_src.defbranch, version = 'main' },
+          { src = repos_src.helptags, version = helptags_rev },
+          { src = repos_src.plugindirs },
+          { src = repos_src.semver, version = vim.version.range('*') },
+        })
+      end)
+
+      local basic_rev = git_get_hash('some-tag', 'basic')
+      local defbranch_rev = git_get_hash('main', 'defbranch')
+      local plugindirs_rev = git_get_hash('HEAD', 'plugindirs')
+      local semver_rev = git_get_hash('v1.0.0', 'semver')
+
+      -- Should properly format as indented JSON
+      local ref_lockfile_lines = {
+        '{',
+        '  "plugins": {',
+        '    "basic": {',
+        '      "rev": "' .. basic_rev .. '",',
+        '      "src": "' .. repos_src.basic .. '",',
+        -- Branch, tag, and commit should be serialized like `'value'` to be
+        -- distinguishable from version ranges
+        '      "version": "\'some-tag\'"',
+        '    },',
+        '    "defbranch": {',
+        '      "rev": "' .. defbranch_rev .. '",',
+        '      "src": "' .. repos_src.defbranch .. '",',
+        '      "version": "\'main\'"',
+        '    },',
+        '    "helptags": {',
+        '      "rev": "' .. helptags_rev .. '",',
+        '      "src": "' .. repos_src.helptags .. '",',
+        '      "version": "\'' .. helptags_rev .. '\'"',
+        '    },',
+        '    "plugindirs": {',
+        '      "rev": "' .. plugindirs_rev .. '",',
+        '      "src": "' .. repos_src.plugindirs .. '"',
+        -- Absent `version` should be missing and not autoresolved
+        '    },',
+        '    "semver": {',
+        '      "rev": "' .. semver_rev .. '",',
+        '      "src": "' .. repos_src.semver .. '",',
+        '      "version": ">=0.0.0"',
+        '    }',
+        '  }',
+        '}',
+      }
+      eq(ref_lockfile_lines, fn.readfile(get_lock_path()))
+    end)
+
+    it('updates lockfile', function()
+      exec_lua(function()
+        vim.pack.add({ repos_src.basic })
+      end)
+      local ref_lockfile = {
+        plugins = {
+          basic = { rev = git_get_hash('main', 'basic'), src = repos_src.basic },
+        },
+      }
+      eq(ref_lockfile, get_lock_tbl())
+
+      n.clear()
+      exec_lua(function()
+        vim.pack.add({ { src = repos_src.basic, version = 'main' } })
+      end)
+
+      ref_lockfile.plugins.basic.version = "'main'"
+      eq(ref_lockfile, get_lock_tbl())
+    end)
+
     it('installs at proper version', function()
       local out = exec_lua(function()
         vim.pack.add({
@@ -1087,6 +1170,20 @@ describe('vim.pack', function()
         local confirm_text = table.concat(api.nvim_buf_get_lines(0, 0, -1, false), '\n')
         matches('Available newer versions:\n• v1%.0%.0\n• v0%.4\n• 0%.3%.1$', confirm_text)
       end)
+
+      it('updates lockfile', function()
+        exec_lua(function()
+          vim.pack.add({ repos_src.fetch })
+        end)
+        local ref_fetch_lock = { rev = hashes.fetch_head, src = repos_src.fetch }
+        eq(ref_fetch_lock, get_lock_tbl().plugins.fetch)
+
+        exec_lua('vim.pack.update()')
+        n.exec('write')
+
+        ref_fetch_lock.rev = git_get_hash('HEAD', 'fetch')
+        eq(ref_fetch_lock, get_lock_tbl().plugins.fetch)
+      end)
     end)
 
     it('works with not active plugins', function()
@@ -1138,6 +1235,9 @@ describe('vim.pack', function()
         '',
       }
       eq(ref_log_lines, vim.list_slice(log_lines, 2))
+
+      -- Should update lockfile
+      eq(git_get_hash('HEAD', 'fetch'), get_lock_tbl().plugins.fetch.rev)
     end)
 
     it('shows progress report', function()
@@ -1350,6 +1450,10 @@ describe('vim.pack', function()
       eq(true, pack_exists('basic'))
       eq(true, pack_exists('plugindirs'))
 
+      local locked_plugins = vim.tbl_keys(get_lock_tbl().plugins)
+      table.sort(locked_plugins)
+      eq({ 'basic', 'plugindirs' }, locked_plugins)
+
       watch_events({ 'PackChangedPre', 'PackChanged' })
 
       n.exec('messages clear')
@@ -1371,6 +1475,23 @@ describe('vim.pack', function()
       eq(3, find_in_log(log, 'PackChangedPre', 'delete', 'plugindirs', nil))
       eq(4, find_in_log(log, 'PackChanged', 'delete', 'plugindirs', nil))
       eq(4, #log)
+
+      -- Should update lockfile
+      eq({ plugins = {} }, get_lock_tbl())
+    end)
+
+    it('works without prior `add()`', function()
+      exec_lua(function()
+        vim.pack.add({ repos_src.basic })
+      end)
+      n.clear()
+
+      eq(true, pack_exists('basic'))
+      exec_lua(function()
+        vim.pack.del({ 'basic' })
+      end)
+      eq(false, pack_exists('basic'))
+      eq({ plugins = {} }, get_lock_tbl())
     end)
 
     it('validates input', function()