Remaining work

Those are the known issues and limitations of the debmans software, serving as an internal, ad-hoc issue tracker.

Blocking

Those are the things that need to be done to complete the restoration of the manpages service.

• setup virtual host configuration, include:
  • redirections for previous links: /man/intro, /man/1/intro, /1/intro, /cgi-bin/man.cgi?query=intro&apropos=1&manpath=Debian+Sid&format=html&locale=en
  • 404 handler to point to 404.html
  • edit cii badge when done, future section
  • CSP and various other SSL flags?
  • CGI config: currently implemented with Flask, see the self-hosted options for a list of deployment possibilities. simplest is probably to provide WSGI and CGI shims. WSGI works well in apache (and also works in Nginx) according digital ocean CGI works everywhere. Also: need a way to pass configs (everything in debmans.search.main() actually) between apache and CGI
• test run on manziarly
  • requires access to manpages group? see RT#6485
  • missing dependencies to run properly (even as plain user): setuptools, click, apt, debian, patch to mirror/debian.org.git
• provide DSA team with Puppet ruleset (see dsa-puppet manifests) or config documentation
• ask DSA to deploy the new code, test
• if it works, fix the manpages.debian.org DNS to point to the static.d.o DNS. at this point, the MVP is in place

Important

Those are not part of the Minimum Viable Product, but would be important to implement to make this software complete.

• search functionality, in that order
  1. whatis(1): find manpages by name (done by Flask app)
  2. apropos(1): find manpages by description
  3. full text search
• i18n: we parse all languages, but should auto-detect the web browser’s language with fallbacks and everything. Apache auto-negociation? Could be like debian.org language menus... The trick is to guess the language list again, similar to the suites issue. The new search box can lookup different locales in the backend, but doesn’t have the list of locales in the frontend.

Nice to have

Those are not really necessary but could improve the service.

• unify site and render? a .mdwn file is like a .1.gz file, basically, except it’s not extracted from a .deb

• add __str__() to classes

• 100% test coverage (about 80% now), edit cii badge when done (quality section)

• move apt_cache optimizations upstream

• rotated –logfile

• add sections browser to the index page?

• debian packaging, edit cii badge when done (future and other sections)

• use tox to test against different py envs

• CII suggestions, edit cii badge when done:

  • continuous integration through Gitlab CI? (quality section)
  • hook pyflakes in test suite (quality section)
  • static code analysis with pylint (analysis section)

• embed test suite in main program

• consider a plugin system for extending to more than manpages, would provide the default for --plugin

  • pluggy: used by py.test, tox and devpi

  • yapsy

  • PluginBase

  • plugnplay

  • click-plugins: relevant only to add new commands

  • SO also suggests using the standard library imp.load_module() or just the builtin __import__(),

    see also the PyPA documentation on how to discover plugins

Possible optimizations

Optimization ideas:

• look at the archive’s Contents-* files to choose which packages to extract
• extract only targeted files from the archive instead of iterating over it? not sure it’s an improvement...
• use os.scandir() where relevant, instead of os.walk() and stat
• use multiprocessor.Pool for background job rendering? maybe by firing up rendering as soon as pages are created
• use md5sums to check if files were modified, edit cii badge when implemented (security section)
• pre-compile all regexes

Those will be implemented as needed, remember:

Premature optimization is the root of all evil. – Donald Knuth

Internal code tasks

Those are todo items extracted from the code. Priority of those is indeterminate unless otherwise noted.

Todo

we should probably have derived Template directly here.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/checkouts/latest/debmans/renderer.py:docstring of debmans.renderer.JinjaRenderer, line 7.)

Todo

support %(target)s instead of standard output, if necessary?

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/checkouts/latest/debmans/renderer.py:docstring of debmans.renderer.CommandRenderer.render, line 6.)

Todo

this assumes cross-references are done with the .Xr macro, which is unfortunately not often the case in my tests. so some manual cross-ref will be required here.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/checkouts/latest/debmans/renderer.py:docstring of debmans.renderer.MandocRenderer, line 3.)

Todo

croaks on the kodi(1) manpage, a weird redirect, which we should handle manually here. the fix, according to mandoc(1) is to chdir to the correct relative directory. looking at zshall(1), .so looks like an “include” directive.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/checkouts/latest/debmans/renderer.py:docstring of debmans.renderer.MandocRenderer, line 7.)

Todo

this may be slow in large directories and may be reimplemented with os.scandir() if we ever depend on Python 3.5 or later.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/checkouts/latest/debmans/renderer.py:docstring of debmans.renderer.find_files, line 4.)

Todo

document that compressed manpages are mandatory

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/debmans/envs/latest/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py:docstring of debmans.renderer.render, line 11.)