Problem: `nvim://` scheme feels more like a generalized interface that
may be requested externally, and it acts like CLI args (roughly).
This is how `vscode://` works.
Anything that behaves like an "app" or a "protocol" deserves its own
scheme. For such Nvim-owned things they will be called `nvim-xx://`.
Solution: Use `nvim-pack://confirm#<bufnr>` template for confirmation
buffer name instead of `nvim://pack-confirm#<bufnr>`.
Problem: Lockfile can become out of sync with what is actually installed
on disk when user performs (somewhat reasonable) manual actions like:
- Delete lockfile and expect it to regenerate.
- Delete plugin directory without `vim.pack.del()`.
- Manually edit lock data in a bad way.
Solution: Synchronize lockfile data with installed plugins on every
lockfile read. In particular:
1. Install immediately all missing plugins with valid lock data.
This helps with "manually delete plugin directory" case by
prompting user to figure out how to properly delete a plugin.
2. Repair lock data for properly installed plugins.
This helps with "manually deleted lockfile", "manually edited
lockfile in an unexpected way", "installation terminated due to
timeout" cases.
3. Remove unrepairable corrupted lock data and their plugins. This
includes bad lock data for missing plugins and any lock data
for corrupted plugins (right now this only means that plugin
path is not a directory, but can be built upon).
Step 1 also improves usability in case there are lazy loaded plugins
that are rarely loaded (like on `FileType` event, for example):
- Previously starting with config+lockfile on a new machine only
installs rare `vim.pack.add()` plugin after it is called (while
an entry in lockfile would still be present). This could be
problematic if there is no Internet connection, for example.
- Now all plugins from the lockfile are installed before actually
executing the first `vim.pack.add()` call in 'init.lua'. And later
they are only loaded on a rare `vim.pack.add()` call.
---
Synchronizing lockfile on its every read makes it work more robustly
if other `vim.pack` functions are called without any `vim.pack.add()`.
---
Performance for a regular startup (good lockfile, everything is
installed) is not affected and usually even increased. The bottleneck
in this area is figuring out which plugins need to be installed.
Previously the check was done by `vim.uv.fs_stat()` for every plugin
in `vim.pack.add()`. Now it is replaced with a single `vim.fs.dir()`
traversal during lockfile sync while later using lockfile data to
figure out if plugin needs to be installed.
The single `vim.fs.dir` approach scales better than `vim.uv.fs_stat`,
but might be less performant if there are many plugins that will be
not loaded via `vim.pack.add()` during startup.
Rough estimate of how long the same steps (read lockfile and normalize
plugin array) take with a single `vim.pack.add()` filled with 43
plugins benchmarking:
- Before commit: ~700 ms
- After commit: ~550 ms
Problem: Currently it is possible to have plugin in a "partial install"
state when `git clone` was successfull but `git checkout` was not.
This was done to not checkout default branch by default in these
situations (for security reasons).
The problem is that it adds complexity when both dealing with lockfile
(plugin's `rev` might be `nil`) and in how `src` and `version` are
treated (wrong `src` - no plugin on disk; wrong `version` - "partial"
plugin on disk).
Solution: Treat plugin as "installed" if both `git clone` and
`git checkout` are successful, while ensuring that not installed
plugins are not on disk and in lockfile.
This also means that if in 'init.lua' there is a `vim.pack.add()` with
bad `version`, for first install there will be an informative error
about it BUT next session will also try to install it. The solution is
the same - adjust `version` beforehand.
Problem: Installation confirmation has several usability issues:
- Choosing "No" results in a `vim.pack.add()` error. This was by
design to ensure that all later code that *might* reference
presumably installed plugin will not get executed. However, this
is often too restrictive since there might be no such code (like
if plugin's effects are automated in its 'plugin/' directory).
Instead the potential code using not installed plugin will throw
an error.
No error on "No" will also be useful for planned lockfile repair.
- List of soon-to-be-installed plugins doesn't mention plugin names.
This might be confusing if plugins are installed under different
name.
Solution: Silently drop installation step if user chose "No" and show
plugin names in confirmation text (together with their pretty aligned
sources).
Problem: Relaxing minimal Git version did not fully preserve previous
behavior in case there no `git` executable. Instead it showed the same
error as if after `vim.system({ 'does_not_exist' })`.
Solution: Show a more direct "No `git` executable" error message.
Problem: No example workflow of how to revert after a bad update.
Solution: Add example workflow of how to revert after a bad update.
In future this might be improved by utilizing other `vim.pack`
features or via a dedicated function (like `vim.pack.restore()` that
restores all installed plugins to a state from the lockfile).
Problem: Changing `src` of an existing plugin cleanly requires manual
`vim.pack.del()` prior to executing `vim.pack.add()` with a new `src`.
Solution: Autodetect `src` change for an existing plugin (by comparing
against lockfile data). If different - properly delete immediately and
treat this as new plugin installation.
Alternative solution might be to update `origin` remote in the
installed plugin after calling `vim.pack.update()`. Although, doable,
this 1) requires more code; and 2) works only for Git plugins (which
might be not the only type of plugins in the future). Automatic
"delete and clean install" feels more robust.
Problem: Plain `vim.pack.add()` calls (with default `opts.load`) does
not fully work if called inside 'plugin/' runtime directory. In
particular, 'plugin/' files of newly added plugins are not sourced.
This is because `opts.load` is `false` during the whole startup, which
means `:packadd!` is used (modify 'runtimepath' but not force source
newly added 'plugin/' files).
This use case is common due to users organizing their config as
separate files in '~/.config/nvim/plugin/'.
Solution: Use newly added `v:vim_did_init` to decide default `opts.load`
value instead of `v:vim_did_enter`.
Problem: Current requirement is Git>=2.36 as `--also-filter-submodules`
flag for `git clone` was introduced there. This is problematic since
default Git version on Ubuntu 22.04 is 2.34.
Solution: Relax minimal Git version to be (at least) 2.0 by selectively
applying necessary flags based on the current Git version.
As 2.0.0 was released in 2014-05-28 (almost the same age as Neovim
project itself), it is reasonable to drop any mention and checks on
minimal version altogether.
Problem: Confirmation buffer is named with `nvim-pack://` as scheme
prefix and uses buffer id (needed for in-process LSP) as one an entry
in the "hierarchical part".
Solution: Use `nvim://pack-confirm#<buf>` format with a more ubiquitous
`nvim://` prefix and buffer id at the end as the optional fragment.
Problem: In some areas plugin's revision is named "state". This might be
confusing for the users.
Solution: Consistently use "revision" to indicate "plugin's state on
disk".
Problem: Using abbreviated version of commit hashes might be unreliable
in the long term (although highly unlikely).
Solution: Use full hashes in lockfile and revision description (in
confirmation buffer and log). Keep abbreviated hashes when displaying
update changes (for brevity).
Problem: `PackChanged[Pre]` events with `kind=update` are triggered both
during plugin's initial installation and after already installed
plugin was updated.
It was a deliberate decision to allow writing only a single update
hook to act as a dedicated "build" entry point (like execute `make` or
`cargo build —release`). This mimics how other plugin managers have a
single "build" command.
This was a result of 'mini.deps' experience with the different
approach: "update" hooks are not run during install. This proved to be
confusing as it requires to write two hooks. But also the reason might
be that 'mini.deps' names it "checkout" hook instead of "update".
However, the `vim.pack` event approach makes it lower cost to handle
separate "update" and "install" events. Something like
`if ev.data.kind == 'install' or ev.data.kind == 'update' then`
instead of two autocommands.
Plus this makes clearer separation of events.
Solution: do not trigger `PackChanged[Pre] kind=update` event during
install.
Problem: Inside `PackChanged[Pre]` callbacks it might be useful to tell
if the affected plugin is active or not. It is already possible via
extra `vim.pack.get({ 'plug-name' })[1].active`, but it is not quite
user-friendly for something that might be needed frequently in real
world use cases.
Solution: Supply extra `active` event data field.
Problem: If plugin was intended to install but there were errors (like
if there is a typo in `src`), lockfile still includes its entry.
This leads to all source of problems (like not correct `get()` output,
not working `update()`, etc.).
Solution: Explicitly account for plugins that were not installed.
Alternative solution might be to write a safe
`lock_set(plug, field, value)` wrapper (which sets field for a correct
`plugins` entry in the lockfile Lua table) and use it after install
to detect the change in `version`. However, this always requires
an extra pass through plugins on every startup, which is suboptimal.
Optimizing for the "happy path" should be a priority in `add()`.
Problem: No way to granularly operate on plugins when inside
confirmation buffer.
Solution: Implement code actions for in-process LSP that act on "plugin
at cursor":
- Update (if has updates).
- Skip updating (if has updates).
- Delete.
Activate via default `gra` or `vim.lsp.buf.code_action()`.
Problem: Installing plugin always pulls latest `version` changes
(usually from the default branch or "latest version tag"). It is more
robust to prefer initial installation to use the latest recorded
(i.e. "working") revision.
Solution: Prefer using revision from the lockfile (if present) during
install. The extra `update()` will pull the latest changes.
Problem: Running `update()` by default doesn't include not active
plugins, because there was no way to get relevant `version` to get
updates from. This might be a problem in presence of lazy loaded
plugins, i.e. ones that can be "not *yet* active" but still needed to
be updated.
Solution: Include not active plugins by default since their `version` is
tracked via lockfile.
Problem: The revision data is returned behind `opts.info` flag because
it required extra Git calls. With lockfile it is not the case.
Solution: Use lockfile to always set `rev` field in output of `get()`.
Problem: `get()` doesn't return `spec.version` about not-yet-active
plugins (because there was no way to know that without `add()`).
Solution: Use lockfile data to set `spec.version` of non-active plugins.
Problem: Some use cases require or benefit from persistent on disk
storage of plugin data (a.k.a. "lockfile"):
1. Allow `update()` to act on not-yet-active plugins. Currently if
`add()` is not yet called, then plugin's version is unknown
and `update()` can't decide where to look for changes.
2. Efficiently know plugin's dependencies without having to read
'pkg.json' files on every load for every plugin. This is for the
future, after there is `packspec` support (or other declaration of
dependencies on plugin's side).
3. Allow initial install to check out the exact latest "working" state
for a reproducible setup. Currently it pulls the latest available
`version.`
4. Ensure that all declared plugins are installed, even if lazy loaded.
So that later `add()` does not trigger auto-install (when there
might be no Internet connection, for example) and there is no issues
with knowing which plugins are used in the config (so even never
loaded rare plugins are still installed and can be updated).
5. Allow `add()` to detect if plugin's spec has changed between
Nvim sessions and act accordingly. I.e. either set new `src` as
origin or enforce `version.` This is not critical and can be done
during `update()`, but it might be nice to have.
Solution: Add lockfile in JSON format that tracks (adds, updtes,
removes) necessary data for described use cases. Here are the required
data that enables each point:
1. `name` -> `version` map.
2. `name` -> `dependencies` map.
3. `name` -> `rev` map. Probably also requires `name` -> `src` map
to ensure that commit comes from correct origin.
4. `name` -> `src` map. It would be good to also track the order,
but that might be too many complications and redundant together
with point 2.
5. Map from `name` to all relevant spec fields. I.e. `name` -> `src`
and `name` -> `version` for now. Storing data might be too much,
but can be discussed, of course.
This commit only adds lockfile tracking without implementing actual
use cases. It is stored in user's config directory and is suggested to
be tracked via version control.
Example of a lockfile:
```json
{
# Extra nesting to more future proof.
"plugins": {
"plug-a": {
"ref": "abcdef1"
"src": "https://github.com/user/plug-a",
# No `version` means it was `nil` (infer default branch later)
},
"plug-b": {
"dependencies": ["plugin-a", "plug-c"],
"src": "https://github.com/user/plug-b",
"ref": "bcdefg2",
# Enclose string `version` in quotes
"version": "'dev'"
},
"plug-c": {
"src": "https://github.com/user/plug-c",
"ref": "cdefgh3",
# Store `vim.version.Range` via its `tostring()` output
"version": ">=0.0.0",
}
}
}
```
Problem: First clean start with config containing multiple
`vim.pack.add()` calls requires to explicitly confirm each one.
Although usually a rare occurrence, it still might be tedious.
Solution: Add a third choice during installation confirmation that
approves current and all next installs within current session. It is
reset after session restart.
Problem: Some environment variables which are useful when working inside
a bare repository can affect any Git operation.
Solution: Explicitly unset problematic environment variables.
Problem:
Force resolve `spec.version` overrides the information about whether
a user supplied `version` or not. Knowing it might be useful in some use
cases (like comparing to previously set `spec` to detect if it has
changed).
Solution:
Do not resolve `spec.version`. This also improves speed when triggering
events and calling `get()`.
- Place default branch first when listing all branches.
- Use correct terminology in `get_hash` helper.
- Do not return `{ '' }` if there are no tags.
Problem:
There is no way to get more information about installed plugins, like
current revision or default branch (necessary if resolving default
`spec.version` manually). As computing Git data migth take some time,
also allow `get()` to limit output to only necessary set of plugins.
Solution:
- introduce arguments to `get(names, opts)`, which follows other
`vim.pack` functions. Plugin extra info is returned by default and
should be opt-out via `opts.info = false`.
- Examples:
- Get current revision: `get({ 'plug-name' })[1].rev`
- Get default branch: `get({ 'plug_name' })[1].branches[1]`
- `update()` and `del()` act on plugins in the same order their names
are supplied. This is less surprising.
- default `opts.info` to `true` since this simplifies logic for the
common user, while still leaving the door open for a faster `get()` if
needed.
Problem: Progress reports use plain `nvim_echo()` with manually
constructed messages and populate history on every call.
Solution: Use `nvim_echo()` with newly added `kind=progress` which (at
least for now) is meant to be a unified interface for showing progress
report. Also save in history only first and last progress report
messages.
Problem:
Nvim does not have a core concept for indicating "progress" of
long-running tasks. The LspProgress event is specific to LSP.
Solution:
- `nvim_echo` can emit `kind="progress"` messages.
- Emits a `Progress` event.
- Includes new fields (id, status, percent) in the `msg_show` ui-event.
- The UI is expected to overwrite any message having the same id.
- Messages have a globally unique ID.
- `nvim_echo` returns the message ID.
- `nvim_echo(… {id=…})` updates existing messages.
Example:
local grp = vim.api.nvim_create_augroup("Msg", {clear = true})
vim.api.nvim_create_autocmd('Progress', {
pattern={"term"},
group = grp,
callback = function(ev)
print(string.format('event fired: %s', vim.inspect(ev))..'\n')
end
})
-- require('vim._extui').enable({enable=true, msg={target='msg', timeout=1000}})
vim.api.nvim_echo({{'searching'}}, true, {kind='progress', percent=80, status='running', title="terminal(ripgrep)"})
local id = vim.api.nvim_echo({{'searching'}}, true, {kind='progress', status='running', percent=10, title="terminal(ripgrep)"})
vim.api.nvim_echo({}, true, {id = id, kind='progress', percent=20, status = 'running', title='find tests'})
vim.api.nvim_echo({}, true, {id = id, kind='progress', status='running', percent=70})
vim.api.nvim_echo({{'complete'}}, true, {id = id, kind='progress', status='success', percent=100, title="find tests"})
Followups:
- Integrate with 'statusline' by listening to the Progress autocmd event.
- Integrate progress ui-event with `vim._extui`.
Problem: Some use cases might lead to `vim.pack.add()` failing to
`:packadd` a plugin because of missing entry in 'packpath'. Like with
`nvim --clean` or manually setting `$XDG_DATA_HOME` during startup.
Solution: Document it. A more proactive approach can be ensuring correct
'packpath' entry, but it is currently somewhat verbose to do (due to
having to adjust for Windows using `\` in 'packpath' entries).
Problem:
The load function in opts was difficult to use if you wished to
customize based on the plugin being loaded.
You could get the name, but without some way to mark a spec, that was of
limited usefulness unless you wanted to hardcode a list of names in the
function, or write a wrapper around the whole thing
Solution:
Allow users to provide an arbitrary data field in plugin specs so that
they may receive info as to how to handle that plugin in load, get() and
events, and act upon it
Co-authored-by: BirdeeHub <birdee@localhost>
Co-authored-by: Evgeni Chasnovski <evgeni.chasnovski@gmail.com>
Problem: No way to skip install confirmation in `add()`. Having install
confirmation by default is a more secure design. However, users are
usually aware of the fact that plugin will be installed and there is
currently no way to skip confirmation.
Plus it can introduce inconvenience on the clean config initialization
if it is modularized with many `vim.pack.add()` calls (leads to
confirming installation many times in a row).
Solution: Add `opts.confirm` option that can skip install confirmation.
Problem: No way to have full control over how plugin is loaded.
Although `:packadd!` has small side effects (only adds plugin
directory to 'runtimepath'; and maybe its 'after/' subdirectory), it
still has side effects. For example, 'plugin/' directories are still
loaded during startup (as part of `:h load-plugins`).
Solution: Allow function `opts.load` that has full control over how
plugin is loaded.
Problem: the `load=true` in `vim.pack.add()` means that `:packadd` is
executed even during startup. This leads to force source of 'plugin/',
which breaks the intended loading order (`:h load-plugins`) and
results into sourcing them twice. This also makes it ignore
`--noplugin` argument.
Using `:packadd!` during startup is more appropriate, while `:packadd`
afterwards is still more favorable to actually force 'plugin/' source
(as there is no pre-defined mechanism that will load them later).
Solution: have `load=false` default during startup, `true` - afterwards.
Problem: Left (`<` prefix) and right (`>` prefix) parts of update
changelog come in not fixed order: it can be left-right or right-left.
Solution: Ensure left-right order.
Problem: New version suggestions in update confirmation buffer might
include semver tags that were committed later but for versions that
are not greater than current. Like if versions committed in order
`v0.2.0` - `v0.3.0` - `v0.2.1` - `v0.3.1`, then when on `v0.3.0` both
`v0.2.1` and `v0.3.1` are suggested, but only the latter is newer as
a version. This is because those tags are computed with post-processed
`git tag --list --contains HEAD`.
Solution: Compute all semver tags and filter only those greater than the
latest version available at HEAD.
Problem: Both `PackChangedPre` and `PackChanged` contain |event-data|
with plugin's `spec`. It looks like a good idea to have all its
triggers contain the same format across all kinds ("install",
"update", "delete"). There are several choices:
- Have it be as verbatim as supplied to `vim.pack.add()`, i.e. can
be either string or table. A bit too ambiguous.
- Have it be table with `src` and `name` inferred. This requires
less work for "install", but more work for "update" and "delete"
(since they use `vim.pack.get()` which already infers default
`version`).
- Have it be table with *all* defaults made explicit. This looks
like the best approach, but requires extra care to only infer
default `version` when needed (i.e. avoid inferring during regular
load) because it is costly in terms of startup time.
This might also introduce inconsistency when dealing with
lockfile(s) as information there should be as close to what user
supplied as possible. Address that when dealing with lockfile.
Solution: Ensure explicit `version` in all events where possible.
Problem: Currently `update()` and `del()` silently ignore input plugin
names that are not for already installed plugin. This might lead to
confusion because they are not explicitly reported.
Solution: Check that all input names are for installed plugins and error
otherwise.
Problem: using `print()` to show progress report writes to `stdout` when
in `--headless` mode (interferes with the testing output) and doesn't
allow coloring.
Solution: use `nvim_echo` with colored chunks.
Problem: Sourcing of 'after/plugin' scripts depends on the value of
'loadplugins' option. It is redundant, as it has effect only during
startup, while it is combined with `vim.v.vim_did_enter == 1` (i.e.
"it is after starting up") condition.
Solution: Ignore it.
Problem: `attempt to yield across metamethod/C-call boundary` error when
trying to use `vim.pack.add()`.
Solution: use `pcall()` variant from 'coxpcall' on non-LuaJIT version of
Lua.
Problem: On canceling the update (triggering `WinClosed`), the tab page
will most probably be closed too. Closing some other tab page while the
confirmation buffer is open also changes tab page numbers. We are trying
to close the wrong tab page in both cases.
Solution: save the tab page ID, and attempt to get the tab page number
from the ID when closing the buffer.
Problem:
Error when adding a plugin will make all following plugins not
`:packadd`ed
Solution:
- add() should handle errors from :packadd with pcall()
Co-authored-by: Evgeni Chasnovski <evgeni.chasnovski@gmail.com>
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
