=================================================== twill: a simple scripting language for web browsing =================================================== .. toctree:: :maxdepth: 2 :caption: Contents: overview install changelog examples browsing commands testing extensions python-api developer other Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. _overview: ============== twill Overview ============== **twill** is a simple language that allows users to browse the Web from a command-line interface. With twill, you can navigate through web sites that use forms, cookies, and most standard Web features. twill supports automated web testing and has a simple Python interface. Check out the :ref:`examples`! twill is open source and written in Python. Downloading twill ----------------- The latest release of twill is twill version 3.4. It is available for `download`_ from the Python Package Index, and you can use Python's `pip`_ tool to install or upgrade twill. Please see the :ref:`changelog` for what's new in this version. twill 3.4 works with Python 3.10 and newer. To start using twill, install it and then type ``twill``. At the prompt, type:: go https://www.slashdot.org/ show showforms showhistory Documentation ------------- The documentation for the latest release is published online at https://twill-tools.github.io/twill/ and https://twill.readthedocs.io/. Documentation is available for the following topics: * :ref:`examples` -- some short examples. * :ref:`browsing` -- General introduction to twill. * :ref:`commands` -- the twill scripting language. * :ref:`testing` -- how to use twill to test Web sites. * :ref:`extensions` -- extension modules that come with twill. * :ref:`python-api` -- for programmers interested in using twill from Python. * :ref:`developer` -- for developers interested in extending or fixing twill. * :ref:`other` -- projects relevant to, or based upon, twill. Contributing ------------ Issues including bug reports, fixes or extensions and pull requests can be submitted on the `GitHub project page`_. When reporting bugs, please be sure to use the ``-f -l debug`` options with ``twill`` so that we can see the full traceback. Authors and License ------------------- The original author of twill is C. Titus Brown, titus@idyll.org. A number of people have contributed bug reports and code since the first release; they are acknowledged below. The original twill source code is Copyright (C) 2005, 2006, 2007 C. Titus Brown. twill is available under the `MIT license`_. Acknowledgements and History ---------------------------- In May 2004, Cory Dodt had a great idea with "Python Browser Poseur" (PBP), his insights helped C. Titus Brown. creating twill. Ian Bicking had the idea of reimplementing PBP on top of IPython (since abandoned in favor of cmd_), and suggested the "in-process" hack. Grig Gheorghiu was strangely enthusiastic about a simple demo shown by Titus and promoted twill since then. John J. Lee has promptly and enthusiastically checked in various patches to mechanize created by Titus. Michele Simionato is an early adopter who has also helped quite a bit. Bug reports have come in from the following fine people: Chris Miles, Matsuno Tokuhiro, Elvelind Grandin, Mike Rovner, sureshvv, Terry Peppers, Kieran Holland, Alexander Shvedunov, Norman Khine, Leonardo Santagada, Sebastien Pierre, Herve Cauwelier, aledain, Uy Do, David Hancock, and Tomi Hautakoski. Patches have been submitted by Joeri van Ruth, Paul McGuire, Ed Rahn, Nic Ferrier, Robert Leftwich, James Cameron, William Volkman, Tommi Virtanen, Simon Buenzli, sureshvv, Jeff Martin, Stephen Thorne, and Bob Halley. Features were proposed by Ben Bangert, and Tristan De Buysscher. In April 2014, Ben Talyor created version 1.8 using requests and lxml instead of mechanize. In July 2016, Christoph Zwerschke created version 2.0 which also integrates ideas and code from Flunc_ which was created by Luke Tucker and Robert Marianski in 2006-2007, and from ReTwill_ which was created in April 2012 as a fork from twill 0.9 by Adam Victor Brandizzi. The official relaunch of twill on its current `GitHub project page`_ and the release of version 2.0 happened in April 2020 with the approval of its original author C. Titus Brown, and Christoph Zwerschke acting as new maintainer of the project. In February 2021, twill 3.0 was released, followed by twill 3.1 in October 2022, and twill 3.2 in October 2023. These versions do not support Python 2 and more, adding type hints and more modernizations. Thanks, all! .. _GitHub project page: https://github.com/twill-tools/twill .. _MIT license: https://opensource.org/licenses/MIT .. _pip: https://pypi.python.org/pypi/pip .. _download: https://pypi.org/project/twill/#files .. _cmd: https://docs.python.org/3/library/cmd.html .. _lxml: http://lxml.de/ .. _requests: http://docs.python-requests.org/ .. _Flunc: https://www.coactivate.org/projects/flunc/project-home .. _Retwill: https://bitbucket.org/brandizzi/retwill/ .. _install: ================ Installing twill ================ The recommended way to install twill is to use pip_:: pip install twill You can also install it directly from a downloaded wheel or source distribution file:: pip install twill-3.4.0.tar.gz Troubleshooting your installation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The first and only thing you should do before creating an issue_ on the `GitHub project page`_ is to see if you have the twill package installed correctly:: python -m twill (This should drop you into the twill shell irrespective of whether 'twill' is on your path.) **If this works**, twill is installed correctly and you just need to make sure that the directory where pip installs scripts is on your path. If you still need help, or the above command doesn't work, please include its output when you create an issue -- thanks! Upgrading twill ~~~~~~~~~~~~~~~ You can use pip to upgrade twill. To get the latest release, use:: pip install -U twill To download the latest development release (which is usually pretty stable) use:: pip install -U https://github.com/twill-tools/twill/archive/main.zip Using twill with AI coding agents ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This documentation is also available as `llms.txt`_ and `llms-full.txt`_, which are suitable for LLMs and AI coding agents. The second file contains the complete documentation in one plain text file. The `twill repository`_ also contains an agent skill in the directory ``skills/twill`` that teaches coding agents how to write, run and debug twill scripts. In Claude Code, you can install it as a plugin with:: /plugin marketplace add twill-tools/twill /plugin install twill@twill-tools For other coding agents, copy the directory ``skills/twill`` into the directory where the agent looks for skills, or use a skill installer that supports GitHub repositories. .. _llms.txt: https://twill-tools.github.io/twill/llms.txt .. _llms-full.txt: https://twill-tools.github.io/twill/llms-full.txt .. _twill repository: https://github.com/twill-tools/twill .. _pip: https://docs.python.org/3/installing/index.html .. _issue: https://github.com/twill-tools/twill/issues .. _GitHub project page: https://github.com/twill-tools/twill .. _changelog: ========= ChangeLog ========= 3.4.0 (released 2026-11-02) --------------------------- * The supported Python versions are now 3.10 to 3.15. * The development dependencies are now defined as dependency groups instead of the optional dependencies 'tests' and 'docs'. * Setting select fields with 'formvalue' has been improved: Options inside optgroups are now found correctly (previously option texts and values could be mismatched), values with whitespace work, exact value matches take precedence, values with a leading '+' or '-' (such as '-1') are first tried as they are, options can be chosen by their label or visible text even if it contains markup, disabled options are ignored like in a browser, and options with duplicate values are handled properly. Note that trying to select a disabled option now raises an error, while previously such an option was silently selected. * Deselecting an option of a single select field with 'formvalue' now clears the selection if this option was selected, so that the first option is used like in a browser. Previously this always tried to set an empty value and failed if there was no such option. * Checked checkboxes and radio buttons with an empty 'value' attribute are now submitted with an empty string like in a browser; 'on' is only sent when the 'value' attribute is missing. Previously, 'on' was also sent for empty values, due to an `lxml bug `_. The same fix applies to the values shown with 'showforms'. Twill now uses its own HTML parser that creates input elements with the fixed behavior. Checkboxes and radio buttons without a 'value' attribute can now also be selected with 'fv' using the default value 'on' when they are part of a group. As with select fields, checkbox values with a leading '+' or '-' are now first tried as they are, and checkboxes with duplicate values are handled properly. * The new command-line option '-s NAME=VALUE' or '--set NAME=VALUE' sets a global variable before the scripts are run, like the 'setglobal' command. This makes it easy to pass values such as an output directory to twill scripts. The command-line options are now also documented. * The new command 'save_content' saves the binary content of the current page, so that downloads such as PDF files can be stored unaltered, which was not possible with 'save_html'. If no filename is given, it is taken from the 'Content-Disposition' header (including RFC 5987 encoded names) or from the URL. Filenames suggested by the server are reduced to their final path component, so that they can never be written outside the current directory, hidden names are ignored, reserved device names on Windows are prefixed with an underscore, and overlong names are truncated. 'save_html' now also respects the 'Content-Disposition' header, and percent-encoded characters in the URL are decoded when constructing a filename. * The new command 'header' asserts that the current page has been sent with a response header whose value matches a regular expression, e.g. 'header content-type ^application/pdf'. The name of the header is case-insensitive, and like with 'url' and 'title', the matching part is stored in the variable '__match__'. The new command 'show_headers' shows all response headers of the current page. * The 'info' command does not fail any more when the current page has been sent without a 'Content-Type' header. * If the first group of a regular expression did not participate in the match, the commands 'find', 'url', 'title' and 'header' now set the variable '__match__' to an empty string instead of None. * The command 'find' with the flag 'x' now also works with XPath expressions that return strings, numbers or booleans instead of elements. * The 'Accept' header sent by default was malformed ('text/html; */*' instead of 'text/html, */*'); this has been fixed. * The command 'extend_with' now warns when an extension command replaces an existing variable. * The documentation of 'follow' now explains how the regular expression is matched against the link text and URL, and the documentation of 'tidy_ok' shows how to allow empty elements such as icons. * The documentation and the help text of 'add_extra_header' now use the correct name of the command (previously given as 'add_extra_headers' or 'add_header'), the documentation of 'form_clear' now shows that it requires the name or number of the form, and the commands 'redirect_error' and 'reset_error' are now documented. * Support for AI coding agents has been added: The documentation now also provides the files 'llms.txt' and 'llms-full.txt', and the repository contains an agent skill for writing and running twill scripts that can be installed as a Claude Code plugin or used with other coding agents. 3.3.1 (released 2025-09-07) --------------------------- * The supported Python versions are now 3.8 to 3.14. 3.3 (released 2024-10-13) ------------------------- * The supported Python versions are now 3.8 to 3.13. * Saving and loading of cookies has been made more robust, and it now also incorporates the domains and paths of the cookies. * Twill now uses Flask_ instead of Quixote_ as test server for running its own test suite. * Twill now uses 'argparse' instead of the deprecated 'optparse' to parse options and arguments passed on the command line. 3.2.5 (released 2024-06-28) --------------------------- * Uses lxml 5 instead of lxml 4. 3.2.4 (released 2024-04-27) --------------------------- * Support latest version of httpx. 3.2.3 (released 2024-03-05) --------------------------- * Adds 'timeout' to the list of exported commends (#22). 3.2.2 (released 2024-01-28) --------------------------- * Fixes the broken interactive shell. * Fixes 'run_file' when specifying files in a subdirectory without extension. 3.2.1 (released 2023-11-23) --------------------------- * Increased the default request timeout of the twill browser to 10 seconds (from 5 seconds in 3.2) and added a command to change the timeout (#18). 3.2 (released 2023-11-02) ------------------------- * The supported Python versions are now 3.8 to 3.12. * A new method 'find_links' was added to the twill browser (#17). * Twill now uses httpx_ instead of requests_. * WSGI apps are now supported via httpx, wsgi_intercept is not needed anymore. * We now use 'pyproject.toml' instead of 'setup.py'. * Type hints and code style have been improved and are checked with ruff. * Internal code was reformatted using ruff format (compatible with black). 3.1 (released 2022-10-30) ------------------------- * The submit command now takes an additional parameter to specify a form that can be used in rare cases when there are no form fields (#7). * Most commands do not return values any more, they are just commands. If you are using twill from Python, you should check browser properties like 'forms' or 'url' instead of using the return values of commands like 'show_forms' or 'back' (see #13, #14). * Two-word commands now consistently have underscores in their names, (e.g. 'form_action', 'get_input', 'show_links'). However, for convenience and backward compatibility, you can still use the names without underscores (e.g. 'formaction', 'getinput', 'showlinks'), and the old two-letter abbreviations (e.g. 'fa' for 'form_action') (#13, #14). * Instead of 'showforms' or 'show_forms' you can now also write 'show forms', and similarly for 'cookies', 'links', 'history' and 'html'. The command 'show html' does the same as 'show' without any arguments. * Renamed shortcuts for user agent strings, and added some more existing ones. * Added type hints (#15). * Support Python 3.11. * Many minor fixes and improvements. 3.0.3 (released 2022-10-12) --------------------------- * Form numbers are now printed correctly with 'showforms' (#12). 3.0.2 (released 2022-04-10) --------------------------- * Save HTML file with browser encoding or as UTF-8 (#9). * Do not modify root logger any more (#10). 3.0.1 (released 2021-12-04) --------------------------- * This version now also supports Python 3.10. * The twill language now allows 8-bit letters to appear in strings unquoted. 3.0 (released 2021-02-25) ------------------------- * In this version we require Python 3.6 to 3.9. If you still need support for Python 2 or Python 3.5, then please use the latest version from the 2.x branch. * The code has been optimized for Python 3 now. * Some minor fixes. 2.0.3 (released 2021-02-25) --------------------------- * Backported the fixes in version 3.0. 2.0.2 (released 2021-02-13) --------------------------- * This version now also supports Python 3.9. * 'tidy_should_exist' has been renamed ot 'require_tidy'. * Support for setting options to be used with HTML Tidy. * Cleanup scripts are now also read as UTF-8 in Python 3. 2.0.1 (released 2020-07-12) --------------------------- * Fixes an issue with encoding declarations (#5). 2.0 (released 2020-04-04) ------------------------- This version is based on twill 1.8, which was a refactoring of version 0.9 that used requests_ and lxml_ instead of mechanize_, done by Ben Taylor in April 2014. It also integrates ideas and code from Flunc_ which was created by Luke Tucker and Robert Marianski in 2006-2007, and from ReTwill_ which was created in April 2012 as a fork from twill 0.9 by Adam Victor Brandizzi. The following improvements and changes were made in this version: * Larger refactoring, clean-up and modernization efforts to support Python 2.7, 3.5 and higher. * The console script has been renamed from 'twill-sh' to just 'twill'. * We assume the default file extension '.twill' for twill scripts now. * Uses lxml_ and requests_ instead of mechanize_ (like in version 1.8), but doesn't need cssselect_ and `Beautiful Soup`_ any more (unlike 1.8). * Removed bundled packages which have become unnecessary (mechanize) or are available in newer versions on PyPI (pyparsing, wsgi_intercept) or in the standard library (subprocess). * Removed parsing options (use_tidy, use_BeautifulSoup, allow_parse_errors) which have become insignificant due to the use of lxml.html. * We use pytest_ instead of nose_ for testing twill now. * A tox_ configuration file for running tests with different Python versions has been added. * Optimized the order of the URLs that are tried out by the twill browser. * Added an option '-d' to dump the last HTML to a file or standard output and an option '-w' to show the HTML directly in the web browser (this feature was taken over from Flunc). * Added alias 'rf' for 'runfiles' and made runfiles run directories of scripts as well. This helps writing test suites for twill scripts. * Added command 'add_cleanup' to unconditionally run cleanup scripts after the current script finished. This allows resetting the state of the tested server, so that tests will always re-run on a clean state. Together with a small init.twill script, this creates a test fixture. (This idea was taken from Flunc, which supports cleanup scripts for test suites, although in a somewhat different way.) * Non string values are now accepted in variable substitution (this feature has been backported from ReTwill). * XPath expressions are now supported in find/notfind commands (this feature has been backported from ReTwill). * Made output better controllable by using log levels (this feature has been backported from ReTwill). See options '-l' and '-o'. * Updated the map of predefined user agent strings. * Basic authentication with realm is now supported again (the 'with_default_realm' option, which was broken in version 1.8, has been switched off). * Server certificates are not verified by default any more, since they are usually not valid on test and staging servers. * Improved handling of meta refresh. Circular redirects are detected and 'debug equiv-refresh' is functional again. A limit for the refresh time interval can be set with the 'equiv_refresh_interval' option. By default this is set to 2, so refresh intervals of 2 or more seconds are ignored. * Moved the examples and additional stuff into an 'extras' directory. * The documentation in the 'docs' directory has been updated and is now created with Sphinx_. * Made sure everything (except twill-forks) also works on Windows. * Fixed a lot of smaller and larger bugs and problems. .. _lxml: https://lxml.de/ .. _requests: https://requests.readthedocs.io/ .. _httpx: https://www.python-httpx.org/ .. _mechanize: https://mechanize.readthedocs.io/ .. _cssselect: https://github.com/scrapy/cssselect .. _Beautiful Soup: https://www.crummy.com/software/BeautifulSoup/ .. _Flunc: https://www.coactivate.org/projects/flunc/project-home .. _Retwill: https://bitbucket.org/brandizzi/retwill/ .. _Sphinx: https://www.sphinx-doc.org/ .. _pytest: https://pytest.org/ .. _nose: https://nose.readthedocs.io/ .. _tox: https://tox.readthedocs.io/ .. _Quixote: https://github.com/nascheme/quixote .. _Flask: https://flask.palletsprojects.com/ .. _examples: ============== twill Examples ============== Example: logging into slashdot ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This script logs you into slashdot (assuming you have an account!):: setlocal username setlocal password go http://www.slashdot.org/ formvalue 1 unickname $username formvalue 1 upasswd $password submit code 200 # make sure form submission is correct! Example: searching Google and going to the first hit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Please be aware that automated searching of Google violates their Terms of Service. This is for example purposes only!*:: setlocal query "twill Python" go https://www.google.com/ fv 1 q $query submit btnI # use the "I'm feeling lucky" button show .. _browsing: ======================= Web browsing with twill ======================= twill strives to be a complete implementation of a web browser, omitting only JavaScript support. It includes support for cookies, basic authentication, and trickery such as "meta refresh" redirects. twill implements a variety of :ref:`commands `. With the built-in language, you can do things like go to a specific URL; follow links; fill out forms and submit them; save, load, and delete cookies; and change the user agent string. You can also easily extend twill with new and specialized commands using Python. .. _commands: commands.html Using twill interactively ~~~~~~~~~~~~~~~~~~~~~~~~~ The twill command line script lets you interactively browse the Web. It features built-in help, e.g. "help go" will describe the command 'go' to you; command-line completion with the TAB key; and history browsing with the UP/DOWN arrow keys. Command-line options ~~~~~~~~~~~~~~~~~~~~ When you pass one or more twill scripts on the command line, twill runs them one after another and reports how many of them succeeded. Without scripts, or with the ``-i`` option, it starts the interactive shell:: usage: twill [-h] [-d DUMPFILE] [-f] [-i] [-l LOGLEVEL] [-n] [-o OUTFILE] [-q] [-s NAME=VALUE] [-u URL] [-v] [-w] [SCRIPT ...] positional arguments: SCRIPT the twill script to execute options: -h, --help show this help message and exit -d, --dump-html DUMPFILE dump HTML to this file on error -f, --fail fail exit on first file to fail -i, --interactive drop into an interactive shell (after running files) -l, --loglevel LOGLEVEL set the logging level -n, --never-fail continue executing scripts past errors -o, --output OUTFILE print log to output file -q, --quiet do not show normal output -s, --set NAME=VALUE set a global variable (can be repeated) -u, --url URL start at the given URL before each script -v, --version show version information and exit -w, --show-error-in-browser show dumped HTML in a web browser Passing values to scripts ~~~~~~~~~~~~~~~~~~~~~~~~~ With ``-s NAME=VALUE`` or ``--set NAME=VALUE`` you can set global variables on the command line, exactly as the 'setglobal' command does. The variables are set before the first script runs, so they can be used in all scripts and in the interactive shell. For instance, if the script ``save.twill`` contains the line ``save_html ${D}/page.html``, you can run it as:: $ twill -s D=/tmp/output save.twill The option can be repeated. The name must be a valid Python identifier that is not also the name of a twill command. The value can be empty and can contain equal signs. There are two alternatives: * Arguments after ``--`` are passed to the scripts as positional arguments. You can load them into the variables ``arg1``, ``arg2``, ... with the :ref:`argparse extension `. * Environment variables can be read via Python's ``os.environ``, since expressions in ``${...}`` are evaluated as Python code:: run "import os" save_html "${os.environ['D']}/page.html" Proxy servers ~~~~~~~~~~~~~ twill understands the ``http_proxy`` environment variable generically used to set proxy server information. To use a proxy in UNIX or Windows, just set the ``http_proxy`` environment variable, e.g.:: % export http_proxy="http://www.someproxy.com:3128" or:: % setenv http_proxy="http://www.someotherproxy.com:3148" Recording scripts ~~~~~~~~~~~~~~~~~ Writing twill scripts is boring. One simple way to get at least a rough initial script is to use the MaxQ_ recorder to generate a twill script. MaxQ_ acts as an HTTP proxy and records all HTTP traffic; I have written a simple twill script generator for it. The script generator and installation docs are included in the twill distribution under the directory ``extras/maxq``. .. _MaxQ: http://maxq.tigris.org/ Miscellaneous implementation details ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * twill ignores robots.txt. * http-equiv=refresh headers are handled immediately, independent of the 'pause' component of the 'content' attribute. * twill ignores JavaScript. Using HTML Tidy to check for broken HTML ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The `HTML Tidy`_ tool can be used to check and clean up broken HTML pages. You can use it with the 'tidy_ok' command to assert that it reports no warnings or errors. In order to use this command in twill, you need to install the `PyTidyLib`_ package. If it is not installed, twill will silently ignore it. It may be desirable to *require* a functioning ``PyTidyLib`` installation; so, to fail when it *isn't* installed, set ``config require_tidy 1``. .. _HTML Tidy: http://www.html-tidy.org/ .. _PyTidyLib: https://pythonhosted.org/pytidylib/ .. _commands: ======================== twill language reference ======================== The following commands are built into twill. Note that all text after a '#' is ignored as a comment, unless it's in a quoted string. Browsing ======== **go** ** -- visit the given URL. **back** -- return to the previous URL. **reload** -- reload the current URL. **follow** ** -- follow the first link on the page that matches the given regular expression. The pattern is searched in the text of each link (including text inside nested markup such as ````) and in its URL, and the first link in the page where either matches is followed. The URL is matched as written in the ``href`` attribute, i.e. relative URLs are not made absolute and percent-escapes such as ``%20`` are kept, but HTML entities are already decoded: a link written as ```` must be matched with ``&`` instead of ``&``, e.g. :: follow "page=2&sort=name" Since the pattern is a regular expression, special characters such as ``?`` or ``.`` must be escaped with a backslash if they shall be matched literally. Note: When used from Python, you get the new URL after executing these commands (after possible redirects) with ``browser.url``. Assertions ========== **code** ** -- assert that the last page loaded had this HTTP status, e.g. ``code 200`` asserts that the page loaded fine. **find** ** [**] -- assert that the page contains the specified regular expression pattern. The variable ``__match__`` is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is also returned. The command also accepts some flags: **i**, for case-insensitive matching, **m** for multi-line mode and **s** for dotall matching. The flag **x** uses XPath expressions instead of regular expressions (see below). **find** ** **x** -- assert that the page contains the element matched by the XPath expression. The variable ``__match__`` is set to the first matching element's text content. When called from Python, the matching text is also returned. **not_find** ** [**] -- assert that the page *does not* contain the specified regular expression pattern. This command accepts the same flags as **find**. Alternative spelling: "notfind" **not_find** ** **x** -- assert that the page *does not* contain the element pointed by the XPath expression. **url** ** -- assert that the current URL matches the given regex pattern. The variable ``__match__`` is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is returned. **title** ** -- assert that the title of this page matches this regular expression. The variable ``__match__`` is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is returned. **header** ** [**] -- assert that the current page has been sent with the given *response* header (the name is case-insensitive) and that its value matches the given regex pattern, e.g. ``header content-type ^application/pdf`` asserts that the page is a PDF document. If the header has been sent several times, its values are joined with ``, ``. Without pattern, only assert that the header is present. The variable ``__match__`` is set to the first matching subgroup (or the entire matching string, if no subgroups are specified, or the entire value, if no pattern is specified). When called from Python, the matching string is returned. Use **show_headers** to see all response headers. Display ======= **echo** ** -- echo the string to the screen. **info** -- display information about the current page. **redirect_output** ** -- append all twill output to the given file. This includes the messages about failed commands. **reset_output** -- display all output to the screen. **redirect_error** ** -- append everything written to the standard error stream to the given file, such as Python warnings or output of **run** commands and extensions. Note that twill's own messages, including the ones about failed commands, are part of the normal output and can be redirected with **redirect_output**. **reset_error** -- write the standard error stream to the screen again. **save_html** *[]* -- save the current page's HTML into a file. If no filename is given, derive the filename like *save_content* does. **save_content** *[]* -- save the binary content of the current page into a file. Use this instead of *save_html* for downloads such as PDF files or spreadsheets, which would be corrupted when saved as text. If no filename is given, it is taken from the ``Content-Disposition`` header sent by the server, or derived from the URL. Filenames suggested by the server or derived from the URL are always reduced to their final path component and saved in the current directory; hidden filenames (starting with a dot) are ignored. If no usable filename can be found, ``download`` with an extension matching the content type is used. *save_html* determines its default filename in the same way, but falls back to ``index.html``. **show** -- show the current page's HTML. Alternative spelling: "showhtml" or "show html" **show_links** -- show all of the links on the current page. Alternative spellings: "showlinks" or "show links" **show_forms** -- show all of the forms on the current page. Alternative spellings: "showforms" or "show forms" **show_history** -- show the browser history. Alternative spellings: "showhistory" or "show history" **show_cookies** -- show the current cookies. Alternative spellings: "showcookies" or "show cookies" **show_headers** -- show the response headers of the current page. Alternative spelling: "show headers" Note: When used from Python, you get these objects using the properties of the browser with the same names, e.g. ``browser.html``, ``browser.links``, ``browser.forms``, ``browser.history`` and ``browser.cookies``. The response headers are available as ``browser.response_headers``. Forms ===== **submit** *[]* -- click the submit button with the given name. Instead of the name of the button you can also specify the number of the form element that shall be used as the submit button. If you do not specify a button name or number, the form is submitted via the last submission button clicked; if nothing had been clicked, use the first submit button on the form. See `details on form handling`_ for more information. In rare cases you may need to specify the form explicitly, you can do so by adding the form name or number as an additional parameter to the button name or number. **form_value** * * --- set the given field in the given form to the given value. For read-only form controls, the click may be recorded for use by **submit**, but the value is not changed unless the 'config' command has changed the default behavior. See 'config' and `details on form handling`_ for more information on the 'form_value' command. For list controls, you can use 'form_value +value' or 'form_value -value' to select or deselect a particular value. Options of select fields can be given by their value, label or visible text. If the given value matches an option or a checkbox as it is, this takes precedence over the meaning of a leading '+' or '-' as a prefix, so that an option or checkbox with the value '-1' can still be selected. Alternative spellings: "formvalue" or "fv" **fv** -- abbreviation for 'form_value'. **form_action** * * -- change the form action URL to the given URL. Alternative spellings: "formaction" or "fa" **fa** -- abbreviation for 'form_action'. **form_clear** ** -- clear all values in the given form. Alternative spelling: "formclear" **form_file** * [ ]* -- attach a file to a file upload button by filename. Alternative spelling: 'formfile'. **show_forms** -- show all of the forms on the current page. Alternative spellings: "showforms" or "show forms" Cookies ======= **save_cookies** ** -- save current cookie jar into a file. **load_cookies** ** -- replace current cookie jar with file contents. **clear_cookies** -- clear all of the current cookies. **show_cookies** -- show all of the current cookies. Alternative spellings: "showcookies" or "show cookies" Debugging ========= **debug** ** ** -- turn on or off debugging/tracing for various functions. The first argument is either 'http' to show HTTP headers, 'equiv-refresh' to show "meta refresh" redirects, or 'commands' to show twill commands. The second argument is '0' for off, '1' for on. Variable handling ================= **set_global** * * -- set variable to value in global dictionary. The value can be retrieved with '$value'. Alternative spelling: "setglobal" **set_local** * * -- set variable to value in local dictionary. The value can be retrieved with '$value'. Alternative spelling: "setlocal" The local dictionary is file-specific, while the global module is general to all the commands. Local variables will override global variables if they exist. Global variables can also be set on the command line with the option ``-s NAME=VALUE``. Note that you can do variable interpolation in strings with ${var}, e.g. :: setglobal a 1 setglobal b 2 fv thisform thatfield "${a}${b}" Other commands ============== **tidy_ok** -- check to see if HTML Tidy runs on this page without any errors or warnings. This check is very stringent, but you can relax the default configuration by setting ``tidy_*`` configuration options with the **config** command. For instance, empty elements such as icons written as ```` cause the warning "trimming empty ". You can allow them with :: config tidy_drop_empty_elements no Any other option of HTML Tidy can be set in the same way, by prefixing its name with ``tidy_`` and replacing hyphens with underscores. **exit** *[]* -- exit with the given integer code, if specified. 'code' defaults to 0. **run** ** -- execute the given Python command. **run_file** * [ ... ]* -- execute the given file(s). Alternative spellings: "runfile" or "rf" **rf** -- abbreviation for 'run_file'. **add_cleanup** * [ ... ]* -- add the given cleanup file(s). These will be run after the current file has executed (successfully or not). **agent** -- set the browser's "User-agent" string. **timeout** *[]* -- set browser timeout to given number of seconds. Defaults to 10 seconds. Set to 0 for no timeout. **sleep** *[]* -- sleep the given number of seconds. Defaults to 1 second. **reset_browser** -- reset the browser. **extend_with** ** -- import commands from Python module. This acts like ``from import *`` does in Python, so e.g. a function ``fn`` in ``extmodule`` would be available as ``fn``. See *extras/examples/extend_example.py* for an example. **get_input** ** -- get keyboard input and store it in ``__input__``. When called from Python, this function returns the input value. Alternative spelling: "getinput" **get_password** ** -- get *silent* keyboard input and store it in ``__password__``. When called from Python, this function returns the input value. Alternative spelling: "getpassword" **add_auth** * * -- add HTTP Basic Authentication information for the given realm/URI combination. For example, :: add_auth IdyllStuff http://www.idyll.org/ titus test tells twill that a request from the authentication realm "IdyllStuff" under ``http://www.idyll.org/`` should be answered with username 'titus', password 'test'. If the 'with_default_realm' option is set to True, ignore 'realm'. **config** [** [**]] -- show/set configuration options. Configuration options starting with ``tidy_`` will be used when checking documents with HTML Tidy. For example, :: config tidy_drop_empty_elements no **add_extra_header** ** ** -- add an extra HTTP header to each HTTP request. **show_extra_headers** -- show the headers being added to each HTTP request. Use **show_headers** to show the headers of the last response instead. **clear_extra_headers** -- clear the headers being added to each HTTP request. Special variables ================= **__input__** -- result of last **getinput**. **__match__** -- matched text from last **find**, **title**, **url** or **header**. **__password__** -- result of last **getpassword**. **__url__** -- current URL. Details on form handling ======================== .. _details on form handling: Both the `form_value` (or `fv`) and `submit` commands rely on a certain amount of implicit cleverness to do their work. In odd situations, it can be annoying to determine exactly what form field `form_value` is going to pick based on your field name, or what form and field `submit` is going to "click" on. Here is the pseudocode for how `form_value` and `submit` figure out what form to use (function `twill.browser.form`):: for each form on page: if supplied regex pattern matches the form name, select if no form name, try converting to an integer N & using N-1 as an index into the list or forms on the page (i.e. form 1 is the first form on the page). Here is the pseudocode for how `form_value` and `submit` figure out what form field to use (function `twill.browser.form_field`):: search current form for control name with exact match to field_name; if single (unique) match, select. if no match, convert field_name into a number and use as an index, if possible. if no match, search current form for control name with regex match to field_name; if single (unique) match, select. if *still* no match, look for exact matches to submit-button values. if single (unique) match, select. Here is the pseudocode for `submit`:: if a form was _not_ previously selected by form_value: if there's only one form on the page, select it. otherwise, fail. if a field is not explicitly named: if a submit button was "clicked" with form_value, use it. otherwise, use the first submit button on the form, if any. otherwise: find the field using the same rules as form_value finally, if a button has been picked, submit using it; otherwise, submit without using a button .. _testing: ============================ Testing Web sites with twill ============================ twill was initially designed for testing web sites, although since then people have also figured out that it's good for :ref:`browsing ` unsuspecting web sites. Using the twill command ~~~~~~~~~~~~~~~~~~~~~~~ The simplest way to test Web sites is to write one or more twill scripts and then simply run :: twill [ -u initial_url ] script(s) either from the command-line (for development purposes), via a cron job (to check to see if sites are up and responding), or from your functional or unit tests (see below). twill will try to run each script given to it on the command line once, and will report the number of scripts that failed. The exit value of the script will be 0 if there are no failures, so you can use it in a shell script easily enough. twill will gather scripts from directories, so you can create a whole directory hierarchy containing your scripts and they will all be gathered and run, in standard lexical order. twill scripts are assumed to have the '.twill' extensions; you don't need to specify it explicitly. The ``-u`` flag can be used to give twill an initial URL; this is equivalent to placing a "go (initial_url)" command at the top of the script, but is particularly handy for test frameworks where the URL might change depending on the developer. Stress testing ~~~~~~~~~~~~~~ You can use the `twill-fork` script to do some stress testing. The syntax is :: twill-fork -n -p script [ scripts... ] For example, :: twill-fork -n 500 -p 10 test-script will fork 10 times and run `test-script` 50 times in each process. `twill-fork` will record the time it takes to run all of the scripts specified on the command and print a summary at the end. The time recorded is *not* the CPU time used. (This would lead to an inaccurate estimate because the client code uses blocking calls to retrieve Web pages.) Rather, the time recorded is the clock time measured between the start and end of script execution. Try `twill-fork -h` to get a list of other command line arguments. Note that twill-fork runs only under Unix and still needs a lot of work... Unit testing ~~~~~~~~~~~~ twill can be used in unit testing, and it contains some Python support infrastructure for this purpose. As an example, here's the code from twill's own unit test, testing the unit-test support code:: import os from .server import app # a Flask app used as test server from .utils import test_dir # directory with test scripts PORT=8090 # port to run the server on def run_server(): """Function to run the server""" app.run(host=HOST, port=PORT) def test(): """The test function""" test_dir = twill.tests.utils.test_dir script = os.path.join(test_dir, 'test_unit_support.twill') # create a TestInfo object test_info = twill.unit.TestInfo(script, run_server, PORT) # run the tests! twill.unit.run_test(test_info) Here, I'm unit testing the Flask_ application ``.server`` in the ``tests`` directory, which is run on the specified ``PORT``, using the twill script ``test_unit_support.twill``. That script contains this code:: # starting URL is provided to it by the unit test support framework. go ./multisubmitform code 200 fv 1 sub_a click submit find "used_sub_a" A few things to note: * the initial URL is set based on the URL reported by ``TestInfo``, which calculates it based on the ``PORT`` argument. (This can be overridden by subclasses.) * ``TestInfo`` contains code to (a) run the server function in a new process, and (b) run the twill script against that server. It then kills the server after script completion. * You can also pass a 'sleep' argument to the ``TestInfo`` constructor that specifies how many seconds to wait for the server to start before executing the script. Testing WSGI applications "in-process" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can pass a WSGI_ application to the ``reset()`` method of the browser. HTTP calls will then go to this application "in-process" directly instead of going over the network. This is particularly useful for unit tests, where setting up an externally available Web server can be inconvenient. For example, the following code redirects all ``localhost:80`` calls to a simple Flask_ app: :: from flask import Flask from twill import browser, commands app = Flask(__name__) @app.route("/") def hello(): return "Hello World!" browser.reset(app=app) commands.go("http://localhost:80") commands.find("Hello World!") See the ``tests/test_wsgi`` unit test for more examples. .. _WSGI: https://peps.python.org/pep-3333/ .. _Flask: https://flask.palletsprojects.com/ .. _extensions: ======================= twill Extension Modules ======================= .. contents:: Several different extension modules are distributed with twill, under 'twill.extensions'. check_links -- a simple link checker ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A simple link checker is included with twill. To use it, do :: go extend_with check_links check_links 'check_links' will visit each link on the current page and verify that the HTTP status code is 200 (success). A list of failing links will be printed out at the end of the function. The 'check_links' function takes an optional argument, a regex to filter for links that should be checked. By default the filter matches everything; if you only wanted to check links on e.g. idyll.org, you could do :: go http://idyll.org/ check_links .*\.idyll\.org A few notes: * 'follow' is used to visit each link, so the referrer URL should be set correctly. * HTTP basic authentication and cookies work as you'd expect, so you can set up twill with all of the permissions necessary to visit restricted links & only then run check_links. match_parse -- extension for regular expressions with multiple matches ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The 'match_parse' extension module contains a number of functions for dealing with multiple matches, etc. Here's some example code: :: extend_with match_parse go http://www.idyll.org/ split "org" echo __matchlist__ findall "t." echo __matchlist__ split "org" popmatch 0 getmatch test 'm[0].split()' showvar test split "org" setmatch "m.split()[0]" popmatch 0 echo __matchlist__ echo __match__ require -- assert that specific conditions hold after each page is loaded ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The 'require' extension module contains four functions that let you assert specific conditions. For example, :: extend_with require require success will assert that after each page load, the HTTP code is 200 ("success"). 'require' does nothing on a 'back' call (which doesn't actually reload a page). Currently there are only two assertions available, 'success' and 'links_ok'. 'links_ok' automatically runs 'check_links' (see above) after each page is loaded. 'links_ok' will not check links twice unless you call 'flush_visited' (see below). The other functions in this module are: **skip_require** -- don't check the requirements for the next action. **no_require** -- turn off requirements checking & reset conditions. **flush_visited** -- flush the list of visited links. mailman_sf -- discard spam messages from your SourceForge mailing lists ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The 'mailman_sf' extension module contains two functions, 'discard_all_messages' and 'exit_if_empty'. Here's some example code: :: # load in the mailman_sf extensions module extend_with mailman_sf # unfortunately we have to hard-code in the mailing list name for # the moment. not sure how to do substitutions here. go https://lists.sourceforge.net/lists/admindb/pywx-announce # fill out the page with the list password. getpassword "Enter list password: " formvalue 1 adminpw __password__ submit 0 code 200 # if there aren't any messages on the page, exit. exit_if_empty # if not empty, discard all messages & submit discard_all_messages submit 0 argparse -- pass arguments into twill scripts via sys.argv ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The 'argparse' extension module contains one function, 'get_args'. 'get_args' loads all of the post-scriptfile command-line arguments into variables, e.g. :: $ twill script1 script2 script3 -- val1 val2 val3 ... >> extend_with argparse >> get_args >> echo "args are ${arg1} ${arg2} ${arg3}" args are val1 val2 val3 dns_check -- make assertions about domain name service records ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ dns_check implements functions to test domain name service (DNS) resolution. You'll need dnspython_ to use it. Functions: * dns_resolves -- assert that a host resolves to a specific IP address. * dns_a -- assert that a host directly resolves to a specific IP address * dns_cname -- assert that a host is an alias for another hostname. * dnx_mx -- assert that a given host is a mail exchanger for the given name. * dns_ns -- assert that a given hostname is a name server for the given name. .. _dnspython: http://www.dnspython.org/ csv_iterate -- iterate over lists of values ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ csv_iterate provides a single function, 'csv_iterate', that lets you run a script once for each row in a file of comma-separated values. For example, if you had a file 'test.csv' containing :: value1, value2 value3, value4 and a script 'test' containing :: echo "first value is ${col1}, second is ${col2}" then :: >> csv_iterate test.csv test would produce :: first value is value1, second is value2 first value is value3, second is value4 This is designed for building test suites containing many different combinations of values for specific forms. dirstack -- manipulate the current working directory (cwd) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ dirstack provides two functions, 'chdir' and 'popd', which change the directory and recover the original directory, respectively. The cwd is kept in the global variable '__dir__'. For example :: >> chdir /tmp >> echo __dir__ >> popd >> echo __dir__ will change to the '/tmp' directory, print out '/tmp', and then change back to the original directory. formfill -- convenience functions for filling out forms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ formfill provides three functions, 'fv_match', 'fv_multi', and 'fv_multi_sub'. 'fv_match' allows you to set many form fields all at once, based on regex matching to the form field name, e.g. :: fv_match
test-(\d+) 'hello world!' will set all fields with the name 'test-#' (# is any number) to 'hello world!' 'fv_multi' allows you to set many form fields at once, using a more compact notation: :: fv_multi field1=value1 field2=value2 ... 'fv_multi' uses the same field-matching convention that 'formvalue' uses: it simply iterates over all arguments, splits at the first '=', and runs 'formvalue' directly. 'fv_multi_sub' does the same thing as 'fv_multi' and then executes 'submit'. (Thanks to Ben Bangert for the idea!) .. _python-api: ================== twill's Python API ================== twill is essentially a web browsing and testing tool based on the `httpx`_ and `lxml`_ packages. All twill commands are implemented in the ``commands.py`` file, and pyparsing_ does the work of parsing the input and converting it into Python commands (see ``parse.py``). Interactive shell work and readline support is implemented via the `cmd`_ module (from the standard Python library). .. _httpx: https://www.python-httpx.org/ .. _lxml: https://lxml.de/ .. _pyparsing: https://github.com/pyparsing/pyparsing .. _cmd: https://docs.python.org/3/library/cmd.html Using twill from Python ~~~~~~~~~~~~~~~~~~~~~~~ There are two fairly simple ways to use twill from Python. (They are compatible with each other, so you don't need to choose between them; just use whichever is appropriate.) The first is to simply import all of the commands in ``commands.py`` and use them directly from Python. For example, :: from twill.commands import * go("http://www.python.org/") showforms() This has the advantage of being very simple, as well as being tied directly to the documented set of commands in :ref:`the commands reference `. However, the functions in ``commands.py`` are too simple for some situations. In particular, most of them do not have any return values, so in order to e.g. get the HTML for a particular page, you will need to ask the actual "Web browser" object that twill uses. To make use of to the Web browser directly, import the ``browser`` object:: from twill import browser browser.go("https://www.python.org/") assert 'Python' in browser.html browser.showforms() This is the second way to use twill from Python, and it is much more flexible. All of the functions in ``commands.py`` use this same browser object, so you can mix and match: :: from twill import browser from twill.commands import * go("https://www.python.org/") assert 'Python' in browser.html find("Documentation") browser.showforms() The basic difference is that functions available through the browser object are intended for use from Python, while ``commands.py`` functions define the twill *language*. In fact, all of the functions in ``commands.py`` are small snippets of code wrapped around the browser object. Most importantly, the browser object also provides several properties that can be used to introspect the current state of the browser and evaluated programmatically in Python, such as ``url``, ``code``, ``html``, ``title``, ``links``, ``forms``, ``cookies``, ``response_headers`` or ``history``. For more information on the functions exposed by the browser object, see the code of the **TwillBrowser** class in twill.browser. Extending twill ~~~~~~~~~~~~~~~ Right now twill is very easy to extend: just build a Python module that exports the functions you want to call, place it in the PYTHONPATH, and run ``extend_with ``. See ``extensions/mailman_sf.py`` for an extension that helps deal with mailman lists on SourceForge; this extension is used by ``examples/discard-sf-mailman-msgs``. Notes: * If your extension raises ``SystemExit``, twill will stop processing the script. This is a useful way to build in conditionals, e.g. see the ``discard-sf-mailman-msgs`` example script. Passing variables from Python into twill ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Suppose you write an extension function, and you want to return some values from that extension function for later use in filling out forms. How do you do this? The simplest way is to simply insert them into the global or local dictionary, like so: :: from twill.namespaces import get_twill_glocals global_dict, local_dict = get_twill_glocals() global_dict['varname'] = value You can then use the usual variable access methods to substitute for varname, e.g. :: formvalue 1 field $varname Using twill in other Python programs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the commands available in twill are implemented as top-level functions in the `twill.commands` module. For example, to use twill functionality from another Python program, you can do:: from twill.commands import go, showforms, formclear, fv, submit go('http://localhost:8080/') go('./widgets') showforms() formclear('1') fv('1', 'name', 'test') fv('1', 'password', 'testpass') fv('1', 'confirm', 'yes') showforms() submit('0') Note that all arguments need to be strings, at least for the moment. You can capture command output by passing any write-enabled file handle to twill.set_output, e.g. :: twill.set_output(StringIO()) will send all non-error output into a StringIO() object. .. _developer: ================ Developing twill ================ twill is entirely written in Python. You will need Python 3.10 or newer to use and develop it. Package tests ~~~~~~~~~~~~~ twill comes with several unit tests. They depend on `pytest`_ and `Flask`_. The development dependencies are defined as dependency groups in ``pyproject.toml``; install them all with ``pip install -e . --group dev`` (requires pip 25.1 or newer). To run the tests, type 'pytest' in the top level directory. To run an individual test, you can use the command ``pytest tests/test_something.py``. To run the tests with all supported Python versions and the code quality checks, run `tox`_. If you use an AI coding agent, it should read the file ``AGENTS.md`` in the top level directory, which describes the project layout, the checks and the conventions for developing twill. .. _pytest: https://pytest.org/ .. _Flask: https://flask.palletsprojects.com/ .. _tox: https://tox.wiki/ Licensing ~~~~~~~~~ twill 0.8 and above are licensed under the `MIT license`_. All code taken from twill 0.x is Copyright (C) 2005-2007 C. Titus Brown . In plain English: C. Titus Brown owns the original code, but you're welcome to use it, update it, subsume it into other projects, and distribute it freely. However, you must retain copyright attribution. Newer versions 1.x, 2.x and 3.x are also Copyright (C) 2007-2026 Ben R. Taylor, Adam V. Brandizzi, Christoph Zwerschke et al. The newer versions are released under the same `MIT license`_. .. _MIT license: http://www.opensource.org/licenses/mit-license.php Developer releases ~~~~~~~~~~~~~~~~~~ "Developer releases" incorporating all recent significant changes are made available at https://github.com/twill-tools/twill.