Enable `SQLITE_JOURNAL_MODE = WAL` for the sqlite integration test
config. With modernc as the default driver, concurrent writers serialize
on SQLite's single write lock and the tail of the queue can exceed the
20s busy timeout under CI load. WAL drains the queue fast enough to stay
inside the timeout (removes rollback's fsync-per-commit and
reader-vs-commit blocking) and covers all sqlite integration tests in
one change.
---
This PR was written with the help of Claude Opus 4.7
---------
Co-authored-by: Claude (Opus 4.7) <noreply@anthropic.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Clean up legacy copied&pasted code, introduce the unique "database
connection" function. Move migration testing helper function
PrepareTestEnv to a separate package.
By the way, remove "shadow connection secrets" tricks: showing
connection string on UI is useless
---------
Co-authored-by: Nicolas <bircni@icloud.com>
* Fix#35685
* Fix#35627
* Fix#31112
Introduce "fipped" config value type, remove unused setting variables.
Make DisableGravatar=true by defult, remove useless config options from
the "Install" page.
The legacy config options are still kept because they are still the
fallback values for the system config options.
---------
Signed-off-by: wxiaoguang <wxiaoguang@gmail.com>
This simplifies the Makefile by removing the whole-file wrapping that
creates a tempdir introduced by
https://github.com/go-gitea/gitea/pull/11126. REPO_TEST_DIR is removed
as well.
Also clean up a lot of legacy code: unnecessary XSS test, incorrect test
env init, unused "_old_uid" hack, etc
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Some minor tooling tweaks:
- Ignore .venv in golangci-lint
- Move go tools to tools directory (e.g. everything that is not "build")
- Enable reportUnusedInlineConfigs in eslint, no current violations
- Apply modernize fix in code-batch-process, modernize cli did
apparently not catch it because of the `go:build ignore` tag.
This PR created a mock webhook server in the tests and added integration
tests for generic webhooks.
It also fixes bugs in package webhooks and pull request comment
webhooks.
The setting `MAILER_TYPE` is deprecated.
According to the config cheat sheet, it should be `PROTOCOL`.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
* Clean the "tools" directory. The "tools" directory contains only two
files, move them.
* The "external_renderer.go" works like "cat" command to echo Stdin to
Stdout , to help testing.
* The `// gobuild: external_renderer` is incorrect, there should be no
space: `//gobuild: external_renderer`
* The `fmt.Print(os.Args[1])` is not a well-defined behavior, and it's
never used.
* The "watch.sh" is for "make watch", it's somewhat related to "build"
* After this PR, there is no "tools" directory, the project root
directory looks slightly simpler than before.
* Remove the legacy "contrib/autoboot.sh", there is no
"gogs_supervisord.sh"
* Remove the legacy "contrib/mysql.sql", it's never mentioned anywhere.
* Remove the legacy "contrib/pr/checkout.go", it has been broken for
long time, and it introduces unnecessary dependencies of the main code
base.
