.. index:: westurner/tools .. index:: Tools .. _tools: ================ Tools ================ | Docs: https://westurner.github.io/tools/ | Src: https://github.com/westurner/tools See Also: https://westurner.github.io/wiki/projects#tools .. index:: Packages .. _packages: Packages ========== | Wikipedia: ``__ A software package is an archive of files with a manifest that lists the files included. Often, the manifest contains file checksums and a *signature*. Many packaging tools make a distinction between source and/or binary packages. Some packaging tools provide configuration options for: * Scripts to run when packaging * Scripts to run at install time * Scripts to run at uninstal time * Patches to apply to the "*vanilla*" source tree, as might be obtained from a version control repository There is a package maintainer whose responsibilities include: * Testing new *upstream* releases * *Vetting* changes from release to release * *Repackaging* upstream releases * *Signing* new package releases *Packaging lag* refers to how long it takes a package maintainer to repackage upstream releases for the target platform(s). .. index:: Anaconda .. _anaconda: Anaconda ~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://store.continuum.io/cshop/anaconda/ | Docs: https://docs.continuum.io/anaconda/ Anaconda is a maintained distribution of :ref:`Conda` packages for many languages; especially :ref:`Python`. .. note:: ``__ (1999) is the installer for :ref:`RPM`-based :ref:`Linux` distributions; which is also written in :ref:`Python` (and :ref:`C`). .. index:: APT .. _apt: APT ~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://alioth.debian.org/projects/apt | Docs: https://wiki.debian.org/Apt | Docs: https://www.debian.org/doc/manuals/debian-reference/ch02.en.html | Docs: https://www.debian.org/doc/manuals/apt-howto/ | Docs: https://wiki.debian.org/SecureApt | Src: git git://anonscm.debian.org/git/apt/apt.git | IRC: ``__ APT ("Advanced Packaging Tool") is the core of :ref:`Debian` package management. * An APT package repository serves :ref:`DEB` packages created with :ref:`Dpkg`. * An APT package repository can be accessed from a local filesystem or over a network protocol ("apt transports") like HTTP, HTTPS, RSYNC, FTP, and BitTorrent (`debtorrent`). An example of APT usage (e.g. to maintain an updated :ref:`Ubuntu` :ref:`Linux` system): .. code-block:: bash apt-get update apt-get upgrade apt-get dist-upgrade apt-cache show bash apt-get install bash apt-get --help man apt-get man sources.list .. index:: AUR .. _aur: AUR ~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Arch_Linux#Arch_User_Repository_.28AUR.29 AUR (:ref:`Arch` *User Repository*) contains :ref:`PKGBUILD` packages which can be installed by :ref:`pacman`. .. index:: Bower .. _bower: Bower ~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.bower.io/ | Src: https://github.com/bower/bower Bower is "a package manager for the web" (:ref:`Javascript` packages) built on :ref:`NPM`. .. index:: BUILD .. _build: BUILD ~~~~~~ | Docs: https://pantsbuild.github.io/build_files.html | Docs: https://pantsbuild.github.io/build_dictionary.html | Docs: https://pantsbuild.github.io/options_reference.html A ``BUILD`` file describes a :ref:`Pants Build` build. .. index:: Cabal .. _cabal: Cabal ~~~~~~ | Homepage: https://www.haskell.org/cabal/ | Docs: https://hackage.haskell.org/ | Docs: https://www.haskell.org/cabal/users-guide/ | Docs: https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/ Cabal is a package manager for :ref:`Haskell` packages. Hackage is the community Cabal package index: https://hackage.haskell.org/ .. index:: Conda Package .. index:: Conda .. _conda: Conda ~~~~~~~ | Wikipedia: ``__ | Docs: https://conda.io/en/latest/ | Src: git https://github.com/conda/conda | PyPI: https://pypi.python.org/pypi/conda Conda is a package build, environment, and distribution system written in :ref:`Python` to install packages written in any language. * Conda was originally created for the Anaconda Python Distribution, which installs packages written in :ref:`Python`, R, :ref:`Javascript`, :ref:`Ruby`, :ref:`C`, :ref:`Fortran` * Conda packages are basically tar archives with build, and optional link/install and uninstall scripts. * ``conda-build`` generates conda packages from conda recipes with a ``meta.yaml``, a ``build.sh``, and/or a ``build.bat``. * Conda recipes reference and build from a source package URI *OR* a :ref:`VCS ` URI and revision; and/or custom ``build.sh`` or ``build.bat`` scripts. * ``conda skeleton`` can automatically create conda recipes from ``PyPI`` (Python), ``CRAN`` (R), and ``CPAN`` (Perl) * ``conda skeleton``-generated recipes can be updated with additional metadata, scripts, and source URIs (as separate patches or consecutive branch commits of e.g. a conda-recipes repository in order to get a diff of the skeleton recipe and the current recipe). * Conda (and :ref:`Anaconda`) packages are hosted by ``__, which hosts free public and paid private Conda packages. * Anaconda Server is an internal "Private, Secure Package Repository" that "supports over 100 different repositories, including PyPI, CRAN, conda, and the Anaconda repository." To create a fresh conda env: .. code:: bash # Python 2.7 conda create -n science --yes python readline conda-env # Python 3.X conda create -n science3 --yes python=3 readline conda-env Work on a conda env: .. code:: bash source activate exmpl2 conda list source deactivate ``conda-env`` writes to and creates environments from ``environment.yml`` files which list conda and :ref:`pip` packages. Work with conda envs and ``environment.yml`` files: .. code:: bash # Install conda-env globally (in the "root" conda environment) conda install -n root conda-env # Create a conda environment with ``conda-create`` and install conda-env conda create -n science python=3 readline conda-env pip # Install some things with conda (and envs/science/bin/pip) # https://github.com/westurner/notebooks/blob/gh-pages/install.sh conda search pandas; conda info pandas conda install blaze dask bokeh odo \ sqlalchemy hdf5 h5py \ scikit-learn statsmodels \ beautiful-soup lxml html5lib pandas qgrid \ ipython-notebook pip install -e git+https://github.com/rdflib/rdflib@master#egg=rdflib pip install arrow sarge structlog # Export an environment.yml #source deactivate conda env export -n science | tee environment.yml # Create an environment from an environment.yml conda env create -n projectname -f ./environment.yml To install a conda package from a custom channel: - https://www.pydanny.com/building-conda-packages-for-multiple-operating-systems.html - https://github.com/conda/conda-recipes/tree/master/cookiecutter - https://binstar.org/pydanny/cookiecutter .. code:: bash conda install -c pydanny cookiecutter # OR pip install cookiecutter The conda-forge custom channel packages are built with :ref:`continuous integration` on multiple platforms: * https://conda-forge.github.io/ * https://anaconda.org/conda-forge Sources: * https://github.com/conda * https://github.com/conda/conda -- conda * https://github.com/ContinuumIO/pycosat -- pycosat SAT solver * https://github.com/conda/conda-env -- conda-env (the ``conda env`` command) * https://github.com/conda/conda-build -- conda-build (the ``conda build`` command) * https://github.com/conda/conda-recipes -- Community-maintained conda recipes (which users may build and :ref:`maintain ` in https://binstar.org package repositories) See also: :ref:`Anaconda`, :ref:`conda-forge` (:ref:`conda-smithy`) .. index:: conda-forge .. _conda-forge: conda-forge ~~~~~~~~~~~~~~~~~ | Homepage: https://conda-forge.github.io/ | Src: https://github.com/conda-forge | Src: https://github.com/conda-forge/feedstocks | Src: https://github.com/conda-forge/feedstocks/tree/master/feedstocks | Src: https://github.com/conda-forge/staged-recipes | DockerHub: https://hub.docker.com/r/condaforge/linux-anvil | Docs: https://conda-forge.github.io/docs/ | Docs: https://conda-forge.github.io/docs/recipe.html * https://conda-forge.github.io/#add_recipe * A. fork: https://github.com/conda-forge/staged-recipes * meta.yaml https://github.com/conda-forge/staged-recipes/blob/master/recipes/example/meta.yaml * B. conda-smithy * meta.yaml * Docs: numpy x.x: https://conda-forge.github.io/docs/meta.html#building-against-numpy * circle.yml * https://github.com/conda-forge/staged-recipes/blob/master/circle.yml * .travis.yml * https://github.com/conda-forge/staged-recipes/blob/master/.travis.yml * appveyor.yml * https://github.com/conda-forge/staged-recipes/blob/master/appveyor.yml * conda-forge.yml * run_docker_build.sh https://github.com/conda-forge/staged-recipes/blob/master/scripts/run_docker_build.sh * bootstrap-obvious-ci-and-miniconda.py https://github.com/conda-forge/conda-smithy/blob/master/bootstrap-obvious-ci-and-miniconda.py .. code:: bash # create a conda package recipe from a pypi package cd $VIRTUAL_ENV/src conda skeleton pypi jupyterthemes ls -ld jupyterthemes/ edit jupyterthemes/meta.yaml # - git repo tags || pypi releases # create a conda-forge feedstock from a conda recipe ## https://github.com/conda-forge/conda-smithy#making-a-new-feedstock cd $VIRTUAL_ENV/src ls -ld jupyterthemes conda-smithy init jupyterhemes ls jupyterthemes-feedstock/ # build a conda-forge feedstock with docker # FROM condaforge/linux-anvil cat ./scripts/run_docker_build.sh ./scripts/run_docker_build.sh ./ci_support/run_docker_build.sh .. index:: conda-smithy .. _conda-smithy: conda-smithy ~~~~~~~~~~~~~~ | Src: https://github.com/conda-forge/conda-smithy | Docs: https://github.com/conda-forge/conda-smithy#making-a-new-feedstock .. index:: DEB .. _deb: DEB ~~~~~ | Wikipedia: ``__ DEB is the :ref:`Debian` software package format. DEB packages are built with :ref:`dpkg` and often hosted in an :ref:`APT` package repository. .. index:: Dpkg .. _dpkg: Dpkg ~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://wiki.debian.org/Teams/Dpkg | Docs: ``_ | Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html | Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkgtools.en.html | Docs: Dpkg is a collection of tools for creating and working with :ref:`DEB` packages. .. index:: dnf .. _dnf: dnf ~~~~~ | Wikipedia: ``__ | Homepage: https://fedoraproject.org/wiki/DNF | Homepage: https://fedoraproject.org/wiki/Features/DNF | Src: git https://github.com/rpm-software-management/dnf | Docs: https://dnf.readthedocs.io/en/latest/ | Docs: https://github.com/rpm-software-management/dnf/wiki | Docs: https://rpm-software-management.github.io/dnf-plugins-core/ dnf is a an open source package manager written in :ref:`Python`. * dnf was introduced in :ref:`Fedora` 18. * dnf is the default package manager in :ref:`Fedora` 22; replacing :ref:`yum`. * [ ] ``yum`` errors if TODO package is installed (* :ref:`salt` provider) * [ ] ``repoquery`` redirects with an error to ``dnf repoquery`` * See ``dnf help`` (and ``man dnf``) * dnf integrates with the Anaconda system installer. * :ref:`dnf` supports **Delta** RPM packages (DRPM), which often significantly reduce the required amount of network transfer required to regularly retrieve and upgrade to the latest repository packages. .. index:: ebuild .. _ebuild: ebuild ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Ebuild | Homepage: https://wiki.gentoo.org/wiki/Ebuild | Docs: https://devmanual.gentoo.org/quickstart/ | Docs: https://devmanual.gentoo.org/ebuild-writing/ | Docs: https://devmanual.gentoo.org/ebuild-writing/file-format/ | Docs: https://devmanual.gentoo.org/ebuild-writing/variables/ | Docs: https://devmanual.gentoo.org/ebuild-writing/use-conditional-code/ | Docs: https://wiki.gentoo.org/wiki/Submitting_ebuilds ebuild is a software package definition format. * ebuilds are like special :ref:`bash` scripts. * ebuilds have ``USE`` flags for specifying build features. * :ref:`Gentoo` is built from ebuild package definitions stored in Gentoo :ref:`Portage`. * :ref:`Portage` packages are built from ebuilds. * The :ref:`emerge` :ref:`Portage` command installs ebuilds. .. index:: emerge .. _emerge: emerge ~~~~~~~~ | Wikipedia: ``__ | Homepage: https://wiki.gentoo.org/wiki/Portage#emerge | Src: git https://github.com/gentoo/portage | Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification | Docs: https://projects.gentoo.org/pms/6/pms.html ``emerge`` is the primary CLI tool used for installing packages built from :ref:`ebuilds ` [from :ref:`Portage`]. .. index:: fpm .. _fpm: fpm ~~~~~ | Wikipedia: | Src: https://github.com/jordansissel/fpm | Docs: https://github.com/jordansissel/fpm/wiki/ | Docs: https://github.com/jordansissel/fpm/wiki/PackageMakeInstall fpm (*effing package management*) is a tool for building many types of software packages from many other types of software packages (e.g. :ref:`DEB`. :ref:`RPM`, :ref:`Python Packages`); often more easily than working with the actual package manager. * fpm package source types include: dir rpm gem python empty tar deb cpan npm osxpkg pear pkgin virtualenv zip. * fpm target package types include: rpm deb solaris puppet dir osxpkg p5p puppet sh tar zip .. index:: Brew .. index:: Homebrew .. _homebrew: Homebrew ~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://brew.sh/ | Src: https://github.com/Homebrew/brew Homebrew is a package manager (``brew``) for :ref:`OSX`. .. index:: NPM .. index:: Node Package Manager .. _npm: NPM ~~~~ | Wikipedia: ``__ | Homepage: https://www.npmjs.org/ | Src: https://github.com/npm/npm | Docs: https://docs.npmjs.com/files/package.json#files NPM is a :ref:`Javascript` package manager created for :ref:`Node.js`. * an NPM package is defined by a ``package.json`` :ref:`JSON` file. * NPM packages are installed with the ``npm`` CLI utility. * :ref:`Bower` builds upon NPM. .. index:: NuGet .. _nuget: NuGet ~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/NuGet | Homepage: https://www.nuget.org/ NuGet is an open source package manager for :ref:`Windows`. * Chocolatey maintains variously updated packages for various windows programs: https://chocolatey.org/ + An example list of Chocolatey NuGet packages as a :ref:`PowerShell` script: https://gist.github.com/westurner/10950476 .. index:: pacman .. _pacman: pacman ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Arch_Linux#Pacman | Homepage: https://www.archlinux.org/pacman/ | Docs: https://wiki.archlinux.org/index.php/Pacman Pacman is an open source package manager which installs ``.pkg.tar.xz`` files for :ref:`Arch` Linux. .. index:: PEX .. _pex: PEX ~~~~~ | Homepage: https://pex.readthedocs.io/ | Src: https://github.com/pantsbuild/pex | PyPI: https://pypi.python.org/pypi/pex | Docs: https://pex.readthedocs.io/en/stable/ PEX (*Python Executable*) is a :ref:`ZIP`-based software package archive format with an executable header. * :ref:`Pants` creates PEX packages. .. index:: PKGBUILD .. _pkgbuild: PKGBUILD ~~~~~~~~~~ | Homepage: https://wiki.archlinux.org/index.php/PKGBUILD | Docs: https://www.archlinux.org/pacman/PKGBUILD.5.html | Docs: https://wiki.archlinux.org/index.php/Makepkg | Docs: https://wiki.archlinux.org/index.php/Creating_packages PKGBUILD is a shell script containing the build information for an :ref:`AUR` :ref:`Arch` :ref:`linux` software package. .. index:: Portage .. _portage: Portage ~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://wiki.gentoo.org/wiki/Project:Portage | Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification | Docs: https://projects.gentoo.org/pms/6/pms.html Portage is a package management and repository system written in :ref:`Python` initially just for :ref:`Gentoo` :ref:`Linux`. * :ref:`Emerge` installs :ref:`ebuilds ` from :ref:`portage`. .. index:: Ports .. _ports: Ports ~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Ports_collection | Homepage: https://www.freebsd.org/ports/ A Ports collection contains *Sources* (e.g. archived releases and patch sets) and :ref:`Makefiles ` designed to compile software :ref:`packages` for particular :ref:`operating systems ` distributions' kernel and standard libraries usually for a particular platform. .. index:: RPM .. _rpm: RPM ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/RPM_Package_Manager RPM (*RPM Package Manager*, :ref:`RedHat` *Package Manager*) is a :ref:`package` format and a set of commandline utilities written in :ref:`C` and :ref:`Perl`. * RPM packages can be installed with ``rpm``, :ref:`yum`, :ref:`dnf`. * RPM pacage can be built with tools like ``rpmbuild`` and ``fpm`` * Python packages can be built into RPM packages with :ref:`setuptools' ` ``bdist_rpm``, ``fpm`` * List contents of RPM packages (archives) with e.g. ``less`` and ``lesspipe``:: less ~/path/to/local.rpm # requires lesspipe to be configured * RPM Packages are served by and retrieved from repositories by tools like :ref:`yum` and :ref:`dnf`: * Local: directories of :ref:`RPM` packages and metadata * Network: :ref:`HTTP `, :ref:`HTTPS `, :ref:`RSYNC`, FTP * :ref:`dnf` supports **Delta** RPM packages (DRPM), which often significantly reduce the required amount of network transfer required to regularly retrieve and upgrade to the latest repository packages. .. note:: There's not yet a :ref:`debtorrent` for :ref:`RPM`, :ref:`YUM`, :ref:`DNF`. .. index:: Egg .. index:: Python Egg .. index:: Python Packages .. _python packages: Python Packages ~~~~~~~~~~~~~~~~~~~~~~~~ | Homepage: https://pypi.python.org/pypi | Download: https://pypi.python.org/simple/ | Docs: https://packaging.python.org/en/latest/ | Docs: https://packaging.python.org/en/latest/current/ | Docs: https://packaging.python.org/en/latest/distributing/ | Docs: https://packaging.python.org/en/latest/peps.html | Docs: https://packaging.python.org/en/latest/projects.html | Docs: https://packaging.python.org/en/latest/specifications/ | Docs: https://pypaio.readthedocs.io/en/latest/roadmap/ A :ref:`Python` Package is a collection of source code and package data files. * Python packages have dependencies: they depend on other packages * Python packages can be served from a package index * :ref:`PyPI` is the community Python Package Index * A Python package is an archive of files (``.zip`` (``.egg``, ``.whl``), ``.tar``, ``.tar.gz``,) containing a ``setup.py`` file containing a version string and metadata that is meant for distribution. * An source dist (``sdist``) package contains source code (every file listed in or matching a pattern in a ``MANIFEST.in`` text file). * A binary dist (``bdist``, ``bdist_egg``, ``bdist_wheel``) is derived from an sdist and may be compiled and named for a specific platform. * sdists and bdists are defined by a ``setup.py`` file which contains a call to a ``distutils.setup()`` or ``setuptools.setup()`` function. * The arguments to the ``setup.py`` function are things like ``version``, ``author``, ``author_email``, and ``homepage``; in addition to package dependency strings required for the package to work (``install_requires``), for tests to run (``tests_require``), and for optional things to work (``extras_require``). * A package dependency string can specify an exact version (``==``) or a greater-than (``>=``) or less-than (``<=``) requirement for each package. * Package names are looked up from an index server (``--index``), such as :ref:`PyPI`, and or an HTML page (``--find-links``) containing URLs containing package names, version strings, and platform strings. * ``easy_install`` (:ref:`setuptools`) and :ref:`pip` can install packages from: the local filesystem, a remote index server, or a local index server. * ``easy_install`` and ``pip`` read the ``install_requires`` (and ``extras_require``) attributes of ``setup.py`` files contained in packages in order to resolve a dependency graph (which can contain cycles) and install necessary packages. . * PyPA Tool Recommendations * | Docs: https://packaging.python.org/en/latest/current * PyPA Python Package PEPs * | Docs: https://packaging.python.org/en/latest/peps.html * PyPA Projects List * | Docs: https://packaging.python.org/en/latest/projects.html .. note:: :ref:`JSON-LD-` for package metadata and environment build metadata could be helpful. - https://github.com/pypa/interoperability-peps/issues/31 .. index:: distutils .. _distutils: Distutils +++++++++ | Docs: https://docs.python.org/2/distutils/ | Docs: https://docs.python.org/3/distutils/ Distutils is a collection of tools for common packaging needs. * Distutils is included in the :ref:`Python` standard library. .. index:: setuptools .. _setuptools: Setuptools ++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Setuptools | Docs: https://pythonhosted.org/setuptools/ | Src: https://github.com/pypa/setuptools | PyPI: https://pypi.python.org/pypi/setuptools Setuptools is a :ref:`Python package ` for working with other :ref:`Python Packages`. * Setuptools builds upon :ref:`distutils` * Setuptools is widely implemented * Most Python packages are installed by setuptools (by :ref:`Pip`) * Setuptools can be installed by downloading ``ez_setup.py`` and then running ``python ez_setup.py``; or, setuptools can be installed with a system package manager (apt, yum) * Setuptools installs a script called ``easy_install`` which can be used to install packages from the local filesystem, a remote index server, a local index server, or an HTML page * ``easy_install pip`` installs :ref:`Pip` from PyPI * Like ``easy_install``, :ref:`Pip` installs python packages, with a number of additional configuration options * Setuptools can build :ref:`RPM` and :ref:`DEB` packages from python packages, with some extra configuration:: python setup.py bdist_rpm --help python setup.py --command-packages=stdeb.command bdist_deb --help .. index:: Pip .. _pip: Pip ++++++++++++++ | Wikipedia: ``_ | Homepage: https://pip.pypa.io/ | Docs: https://pip.pypa.io/en/latest/user_guide.html | Docs: https://pip.readthedocs.io/en/latest/ | Src: git https://github.com/pypa/pip | Pypi: https://pypi.python.org/pypi/pip | IRC: #pypa | IRC: #pypa-dev Pip is a tool for installing, upgrading, and uninstalling :ref:`Python` packages. :: pip help pip help install pip --version sudo apt-get install python-pip pip install --upgrade pip pip install libcloud pip install -r requirements.txt pip uninstall libcloud * Pip stands upon :ref:`distutils` and :ref:`setuptools`. * Pip retrieves, installs, upgrades, and uninstalls packages. * Pip can list installed packages with ``pip freeze`` (and ``pip list``). * Pip can install packages as 'editable' packages (``pip install -e``) from version control repository URLs which must begin with ``vcs+``, end with ``#egg=``, and may contain an ``@vcstag`` tag (such as a branch name or a version tag). * Pip installs packages as editable by first cloning (or checking out) the code to ``./src`` (or ``${VIRTUAL_ENV}/src`` if working in a :ref:`virtualenv`) and then running ``setup.py develop``. * Pip configuration is in ``${HOME}/.pip/pip.conf``. * Pip can maintain a local cache of downloaded packages, which can lessen the load on package servers during testing. * Pip skips reinstallation if a package requirement is already satisfied. * Pip requires the ``--upgrade`` and/or ``--force-reinstall`` options to be added to the ``pip install`` command in order to upgrade or reinstall. * At the time of this writing, the latest stable pip version is ``1.5.6``. .. warning:: With :ref:`Python` 2, pip is preferable to :ref:`setuptools`'s ``easy_install`` because pip installs ``backports.ssl_match_hostname`` in order to validate ``HTTPS`` certificates (by making sure that the certificate hostname matches the hostname from which the DNS resolved to). Cloning packages from source repositories over ``ssh://`` or ``https://``, either manually or with ``pip install -e`` avoids this concern. There is also a tool called :ref:`peep` which requires considered-good SHA256 checksums to be specified for every dependency listed in a ``requirements.txt`` file. For more information, see: https://legacy.python.org/dev/peps/pep-0476/#python-versions .. glossary:: Pip Requirements File Plaintext list of packages and package URIs to install. Requirements files may contain version specifiers (``pip >= 1.5``) Pip installs Pip Requirement Files:: pip install -r requirements.txt pip install --upgrade -r requirements.txt pip install --upgrade --user --force-reinstall -r requirements.txt An example ``requirements.txt`` file:: # install pip from the default index (PyPI) pip --index=https://pypi.python.org/simple --upgrade pip # Install pip 1.5 or greater from PyPI pip >= 1.5 # Git clone and install pip as an editable develop egg -e git+https://github.com/pypa/pip@1.5.X#egg=pip # Install a source distribution release from PyPI # and check the MD5 checksum in the URL https://pypi.python.org/packages/source/p/pip/pip-1.5.5.tar.gz#md5=7520581ba0687dec1ce85bd15496537b # Install a source distribution release from Warehouse https://warehouse.python.org/packages/source/p/pip/pip-1.5.5.tar.gz # Install an additional requirements.txt file -r requirements/more-requirements.txt .. index:: Peep .. _peep: Peep +++++ | Src: https://github.com/erikrose/peep | PyPI: https://pypi.python.org/pypi/peep Peep works just like :ref:`pip`, but requires ``SHA256`` checksum hashes to be specified for each package in ``requirements.txt`` file. .. index:: Python Package Index .. index:: PyPI .. _pypi: PyPI ++++++ | Wikipedia: https://en.wikipedia.org/wiki/Python_Package_Index | Docs: https://wiki.python.org/moin/CheeseShop | Docs: https://wiki.python.org/moin/CheeseShopDev | Homepage: https://pypi.python.org/pypi | Src: https://github.com/pypa/pypi-legacy PyPI-legacy is the original Python Package Index. PyPI is now powered by :ref:`Warehouse` .. index:: Warehouse .. _warehouse: Warehouse ++++++++++ | Homepage: https://warehouse.python.org/ | Src: https://github.com/pypa/warehouse | Docs: https://warehouse.readthedocs.io/en/latest/ Warehouse is the "Next Generation Python Package Repository". All packages uploaded to :ref:`PyPI` are also available from Warehouse. .. index:: Devpi .. _devpi: Devpi ++++++++ | Homepage: https://doc.devpi.net/ | Src: https://github.com/devpi/devpi | Issues: https://github.com/devpi/devpiissues | PyPI: https://pypi.python.org/pypi/devpi-server | PyPI: https://pypi.python.org/pypi/devpi-web | PyPI: https://pypi.python.org/pypi/devpi-client | Docs: https://doc.devpi.net/latest/ Devpi is a server and client solution for :ref:`Python package ` mirroring, hosting, and testing. .. index:: Python Wheel .. index:: Wheel .. _wheel: Wheel ++++++ | Docs: https://legacy.python.org/dev/peps/pep-0427/ | Docs: https://wheel.readthedocs.io/en/latest/ | Src: https://github.com/pypa/wheel | PyPI: https://pypi.python.org/pypi/wheel * Wheel is a newer, PEP-based standard (``.whl``) with a different metadata format, the ability to specify (JSON) digital signatures for a package within the package, and a number of additional speed and platform-consistency advantages. * Wheels can be uploaded to PyPI. * Wheels are generally faster than traditional Python packages. Packages available as wheels are listed at ``__. .. index:: Ruby Gem .. index:: RubyGems .. _rubygems: RubyGems ~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/RubyGems | Homepage: https://rubygems.org/ | Docs: https://guides.rubygems.org/ | Src: https://github.com/rubygems/rubygems RubyGems is a package manager for :ref:`Ruby` packages ("Gems"). .. index:: Yum .. _yum: Yum ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Yellowdog_Updater,_Modified | Homepage: http://yum.baseurl.org/ Yum is a tool for installing, upgrading, and uninstalling :ref:`RPM` packages. .. index:: Version Control Systems .. index:: Distributed Version Control Systems .. _vcs: Version Control Systems ======================== | Wikipedia: https://en.wikipedia.org/wiki/Revision_control | Wikipedia: https://en.wikipedia.org/wiki/Distributed_revision_control Version Control Systems (VCS) --- or Revision Control Systems (RCS) --- are designed to solve various problems in change management. * VCS store code in a **repository**. * Changes to one or more files are called **changesets**, **commits**, or **revisions** * Changesets are **comitted** or **checked into** to a repository. * Changesets are **checked out** from a repository * Many/most VCS differentiate between the repository and a **working directory**, which is currently **checked out** to a specific *changeset* identified by a **revision identifier**; possibly with **uncommitted** local changes. * A **branch** is forked from a line of development and then **merged** back in. * Most projects designate a *main line* of development referred to as a **trunk**, **master**, or **default** branch. * Many projects work with *feature* and *release* branches, which, ideally, eventually converge by being merged back into **trunk**. (see: :ref:`HubFlow` for an excellent example of branching) * Traditional VCS are centralized on a single point-of-failure. * Some VCS have a concept of *locking* to prevent multiple peoples' changes from *colliding* * Distributed Version Control Systems (DVCS) (can) **clone** all **revisions** of every **branch** of a repository every time. * * DVCS changesets are **pushed** to a different repository * DVCS changesets are **pulled** from another repository into a *local* **clone** or **copy** of a repository * Teams working with DVCS often designate a central repository hosted by a project forge service like SourceForge, GNU Savannah, GitHub, or BitBucket. * Contributors send **patches** which build upon a specific revision, which can be applied by a maintainer with **commit access** permissions. * Contributors **fork** a new **branch** from a specific revision, commit changes, and then send a **pull request**, which can be applied by a maintainer with **commit access** permissions. .. index:: CVS .. _cvs: CVS ~~~~~ | Homepage: https://www.nongnu.org/cvs/ | Homepage: https://savannah.nongnu.org/projects/cvs | Wikipedia: https://en.wikipedia.org/wiki/Concurrent_Versions_System | Docs: https://www.nongnu.org/cvs/#documentation CVS (``cvs``) is a centralized version control system (VCS) written in :ref:`C`. CVS predates most/many other VCS. .. index:: Subversion .. _subversion: Subversion ~~~~~~~~~~~~~ | Homepage: https://subversion.apache.org/ | Wikipedia: https://en.wikipedia.org/wiki/Apache_Subversion | Docs: https://subversion.apache.org/docs/ | Docs: https://subversion.apache.org/quick-start | Src: svn https://svn.apache.org/repos/asf/subversion/trunk | Src: https://github.com/apache/subversion Apache Subversion (``svn``) is a centralized revision control system (VCS) written in :ref:`C`. To checkout a revision of a repository with ``svn``: .. code:: bash svn co https://svn.apache.org/repos/asf/subversion/trunk subversion .. index:: Bazaar .. _bazaar: Bazaar ~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNU_Bazaar | Homepage: https://bazaar.canonical.com/en/ | Homepage: https://launchpad.net/bzr | Docs: http://doc.bazaar.canonical.com/en/ | Docs: http://doc.bazaar.canonical.com/latest/en/mini-tutorial/index.html | Src: bzr lp:bzr GNU Bazaar (``bzr``) is a distributed revision control system (DVCS, RCS, VCS) written in :ref:`Python` and :ref:`C`. https://launchpad.net hosts Bazaar repositories; with special support from the ``bzr`` tool in the form of ``lp:`` URIs like ``lp:bzr``. To clone a repository with ``bzr``: .. code:: bash bzr branch lp:bzr .. index:: Git .. _git: Git ~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://git-scm.com/ | Docs: https://git-scm.com/documentation | Docs: https://git-scm.com/book/en/ | Docs: https://documentup.com/skwp/git-workflows-book | Docs: https://learnxinyminutes.com/docs/git/ | Src: git https://github.com/git/git Git (``git``) is a distributed version control system for tracking a branching and merging repository of file revisions written in :ref:`C` (DVCS, VCS, RCS). To clone a repository with ``git``: .. code:: bash git clone https://github.com/git/git .. index:: GitFlow .. _gitflow: GitFlow ~~~~~~~~~ | Src: https://github.com/nvie/gitflow | Docs: https://nvie.com/posts/a-successful-git-branching-model/ | Docs: https://github.com/nvie/gitflow/wiki | Docs: https://github.com/nvie/gitflow/wiki/Command-Line-Arguments | Docs: https://github.com/nvie/gitflow/wiki/Config-values GitFlow is a named branch workflow for :ref:`git` with ``master``, ``develop``, ``feature``, ``release``, ``hotfix``, and ``support`` branches (``git flow``). Gitflow branch names and prefixes are configured in ``.git/config``; the defaults are: .. table:: GitFlow Branch Names :class: table-striped table-responsive +--------------------+-------------------------------------------------------------------------------+ | **Branch Name** | **Description** | | | (and `Code Labels `__) | +--------------------+-------------------------------------------------------------------------------+ | ``master`` | Stable trunk (latest release) | +--------------------+-------------------------------------------------------------------------------+ | ``develop`` | Development main line | +--------------------+-------------------------------------------------------------------------------+ | ``feature/`` | New features for the next release (e.g. ``ENH``, ``PRF``) | +--------------------+-------------------------------------------------------------------------------+ | ``release/`` | In-progress release branches (e.g. ``RLS``) | +--------------------+-------------------------------------------------------------------------------+ | ``hotfix/`` | Fixes to merge to both ``master`` and ``develop`` | | | (e.g. ``BUG``, ``TST``, ``DOC``) | +--------------------+-------------------------------------------------------------------------------+ | ``support/`` | "What is the 'support' branch?" | | | | | | https://github.com/nvie/gitflow/wiki/FAQ | +--------------------+-------------------------------------------------------------------------------+ Creating a new release with :ref:`Git` and :ref:`GitFlow`: .. code:: bash git clone ssh://git@github.com/westurner/dotfiles # git checkout master # git checkout -h # git help checkout (man git-checkout) # git flow [ -h] # git-flow [ -h] git flow init ## Update versiontag in .git/config to prefix release tags with 'v' git config --replace-all gitflow.prefix.versiontag v cat ./.git/config # [gitflow "prefix"] # feature = feature/ # release = release/ # hotfix = hotfix/ # support = support/ # versiontag = v # ## feature/ENH_print_hello_world git flow feature start ENH_print_hello_world #git commit, commit, commit git flow feature git flow feature finish ENH_print_hello_world # ENH ## release/0.1.0 git flow release start 0.1.0 #git commit (e.g. update __version__, setup.py, release notes) git flow release finish 0.1.0 git flow release finish 0.1.0 git tag | grep 'v0.1.0' .. index:: HubFlow .. _hubflow: HubFlow ~~~~~~~~~ | Src: https://github.com/datasift/gitflow | Docs: https://datasift.github.io/gitflow/ | Docs: https://datasift.github.io/gitflow/IntroducingGitFlow.html | Docs: https://datasift.github.io/gitflow/TheHubFlowTools.html | Docs: https://datasift.github.io/gitflow/GitFlowForGitHub.html GitFlow is a named branch workflow for :ref:`git` with ``master``, ``develop``, ``feature``, ``release``, ``hotfix``, and ``support`` branches (``git flow``). HubFlow is a fork of :ref:`GitFlow` that adds useful commands for working with :ref:`Git` and GitHub **pull requests**. HubFlow branch names and prefixes are configured in ``.git/config``; the defaults are: .. table:: HubFlow Branch Names :class: table-striped table-responsive +--------------------+-------------------------------------------------------------------------------+ | **Branch Name** | **Description** | | | (and `Code Labels `__) | +--------------------+-------------------------------------------------------------------------------+ | ``master`` | Stable trunk (latest release) | +--------------------+-------------------------------------------------------------------------------+ | ``develop`` | Development main line | +--------------------+-------------------------------------------------------------------------------+ | ``feature/`` | New features for the next release (e.g. ``ENH``, ``PRF``) | +--------------------+-------------------------------------------------------------------------------+ | ``release/`` | In-progress release branches (e.g. ``RLS``) | +--------------------+-------------------------------------------------------------------------------+ | ``hotfix/`` | Fixes to merge to both ``master`` and ``develop`` | | | (e.g. ``BUG``, ``TST``, ``DOC``) | +--------------------+-------------------------------------------------------------------------------+ Creating a new release with :ref:`Git` and :ref:`HubFlow`: .. code:: bash git clone ssh://git@github.com/westurner/dotfiles # git checkout master # git checkout -h # git help checkout (man git-checkout) # git hf [ -h] # git-hf [ -h] git hf init ## Update versiontag in .git/config to prefix release tags with 'v' git config --replace-all hubflow.prefix.versiontag v #cat .git/config # ... # [hubflow "prefix"] # feature = feature/ # release = release/ # hotfix = hotfix/ # support = support/ # versiontag = v # git hf update git hf pull git hf pull -h ## feature/ENH_print_hello_world git hf feature start ENH_print_hello_world #git commit, commit git hf pull git hf push #git commit, commit git hf feature finish ENH_print_hello_world # ENH ## release/0.1.0 git hf release start 0.1.0 ## commit (e.g. update __version__, setup.py, release notes) git hf release finish 0.1.0 git hf release finish 0.1.0 git tag | grep 'v0.1.0' The GitFlow HubFlow illustrations are very helpful for visualizing and understanding any DVCS workflow: ``__. .. figure:: https://datasift.github.io/gitflow/GitFlowMasterBranch.png :alt: GitFlow Release / Master Branch Merge Diagram :target: https://datasift.github.io/gitflow/IntroducingGitFlow.html .. figure:: https://datasift.github.io/gitflow/GitFlowHotfixBranch.png :alt: GitFlow Hotfix to Master and Develop Branches Merge Diagram :target: https://datasift.github.io/gitflow/IntroducingGitFlow.html .. figure:: https://datasift.github.io/gitflow/GitFlowWorkflowNoFork.png :alt: Numbered GitFlow Workflow Diagram :target: https://datasift.github.io/gitflow/GitFlowForGitHub.html .. index:: Hg .. index:: Mercurial .. _hg: .. _mercurial: Mercurial ~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Mercurial | Homepage: https://www.mercurial-scm.org/ | Docs: https://www.mercurial-scm.org/guide | Docs: https://book.mercurial-scm.org/ | Src: hg https://www.mercurial-scm.org/repo/hg | Src: hg http://hg.intevation.org/mercurial Mercurial (``hg``) is a distributed revision control system written in :ref:`Python` and :ref:`C` (DVCS, VCS, RCS). To clone a repository with ``hg``: .. code:: bash hg clone https://www.mercurial-scm.org/repo/hg .. index:: Project Templates .. _project templates: Project Templates =================== * :ref:`cookiecutter` (Python, [...]) * :ref:`yeoman` (JS, [...]) * :ref:`jinja2` (:ref:`salt`, :ref:`ansible` :ref:`configuration management` .. index:: Exemplar Projects .. _exemplar projects: Exemplar Projects ~~~~~~~~~~~~~~~~~~~~~~ * https://www.python.org/dev/peps/ | "PEP 0 -- Index of Python Enhancement Proposals (PEPs)" * https://www.python.org/dev/peps/pep-0012/ | "PEP 0012 -- Sample :ref:`reStructuredText` PEP Template" (:ref:`Python`) * https://github.com/ipython/ipython/wiki/IPEPs:-IPython-Enhancement-Proposals | "IPEPs: IPython Enhancement Proposals" * https://github.com/ipython/ipython/wiki/IPEP-0%3A-IPEP-Template | "IPEP 0: IPEP Template" (:ref:`IPython`, :ref:`Jupyter`) * https://github.com/jupyter/roadmap * github.com/westurner/wiki: a GitHub Sphinx Wiki | Home: https://westurner.github.io/wiki/ | Wiki: https://github.com/westurner/wiki/wiki | Src: https://github.com/westurner/wiki | Src: https://github.com/westurner/wiki.wiki.git | Issues: https://github.com/westurner/wiki/issues .. code:: bash make help # - build the docs w/ sphinx # - push to both branches # - gh-pages from eg _build/html, _build/singlehtml make docs push gh-pages * github.com/rdfjs/rdfjs.org: a GitHub Project | Home: https://github.com/rdfjs/rdfjs.org | Wiki: https://github.com/rdfjs/rdfjs.org/wiki | Src: https://github.com/rdfjs/rdfjs.org | Src: https://github.com/rdfjs/rdfjs.org.wiki.git | Issues: https://github.com/rdfjs/rdfjs.org/issues * https://github.com/rdfjs/rdfjs.org/wiki/Meetings seeAlso: * https://wrdrd.github.io/docs/consulting/software-development#agile * https://wrdrd.github.io/docs/consulting/team-building#the-same-page .. index:: cookiecutter .. _cookiecutter: cookiecutter ~~~~~~~~~~~~~ | Homepage: https://github.com/audreyr/cookiecutter | Src: git https://github.com/audreyr/cookiecutter | PyPI: https://pypi.python.org/pypi/cookiecutter | Docs: https://cookiecutter.readthedocs.io/en/latest/ | Docs: https://cookiecutter.readthedocs.io/en/latest/usage.html | Docs: https://cookiecutter.readthedocs.io/en/latest/tutorials.html#create-your-very-own-cookiecutter Cookiecutter creates projects (files and directories) from project templates written in :ref:`Jinja2` for projects written in :ref:`Python` and other languages. * https://cookiecutter.readthedocs.io/en/latest/readme.html#available-cookiecutters List of Cookiecutter generators * https://github.com/audreyr/cookiecutter-pypackage Cookiecutter template for a :ref:`Python Package ` (e.g. ``setup.py``, ``docs/``, ``README.rst``) * https://github.com/pydanny/cookiecutter-django Cookiecutter template for a Django project w/ Bootstrap, AngularJS, :ref:`Docker`, * see also: https://github.com/xenith/django-base-template * https://github.com/pydanny/cookiecutter-djangopackage Cookiecutter template for reusable Django packages (**installable apps**). * https://github.com/openstack-dev/cookiecutter Cookiecutter template for :ref:`OpenStack` :ref:`Python Package ` projects (\* :ref:`pip` \*, pbr, tox, :ref:`sphinx`) * https://github.com/openstack-dev/specs-cookiecutter Cookiecutter template for :ref:`OpenStack` specs projects (pbr, tox, :ref:`sphinx`) .. index:: yeoman .. _yeoman: yeoman ~~~~~~~~ | Wikipedia: ``__ | Homepage: https://yeoman.io/ | Src: git https://github.com/yeoman/yeoman | Src: https://github.com/yeoman | NPM: https://www.npmjs.com/package/yo | NPMPkg: ``yo`` | Docs: https://yeoman.io/learning/ | Docs: https://yeoman.io/codelab/ | Docs: https://yeoman.io/authoring | Docs: https://yeoman.io/learning/resources.html * https://github.com/yeoman/generator-generator Generate a Yeoman generator (``./authoring``). * https://yeoman.io/generators/ List of Yeoman generators * https://github.com/yeoman/generator-angular AngularJS 1 Yeoman generator (:ref:`bower`, karma tests, :ref:`CoffeeScript`, :ref:`TypeScript` ) * https://github.com/diegonetto/generator-ionic Apache Cordova mobile app w/ Ionic (AngularJS 1, :ref:`grunt`) * https://stackoverflow.com/questions/29649578/available-yeoman-generator-for-angular-2 AngularJS 2 Yeoman generators (:ref:`TypeScript`) * https://github.com/kriasoft/react-starter-kit React.js, Express, Flux, ES6+, JSX, Babel, PostCSS, Webpack, BrowserSync (:ref:`Node.js`) * https://github.com/yeoman/generator-polymer Polymer Web Components (:ref:`gulp`) .. index: Languages .. _languages: Languages ======================= .. index: Programming Languages .. _programming languages: .. _programming language: Programming Languages ~~~~~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Programming_language | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_languages | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_language_classification * https://en.wikipedia.org/wiki/Von_Neumann_programming_languages * https://en.wikipedia.org/wiki/Von_Neumann_architecture#Mitigations * https://en.wikipedia.org/wiki/NX_bit .. contents: :local: .. index:: Programming Paradigms .. _programming paradigms: Programming Paradigms ~~~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Programming_paradigm | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Programming_paradigms * https://en.wikipedia.org/wiki/Comparison_of_programming_paradigms .. index:: Lightweight Markup Languages .. _lightweight markup language: Lightweight Markup Language ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Lightweight_markup_language | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Lightweight_markup_languages .. index:: BBCode .. _bbcode: BBCode ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/BBCode | Homepage: https://www.bbcode.org/ | Docs: https://www.bbcode.org/reference.php | Docs: https://www.bbcode.org/examples/ BBCode is a :ref:`Lightweight markup language` often used by bulletin boards and forums. .. index:: Markdown .. _markdown: Markdown ++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Markdown | Homepage: https://daringfireball.net/projects/markdown/ | Standard: https://daringfireball.net/projects/markdown/syntax | Docs: https://www.w3.org/community/markdown/wiki/MarkdownImplementations | Docs: https://en.wikipedia.org/wiki/Markdown#Implementations | Docs: https://learnxinyminutes.com/docs/markdown/ | Docs: https://guides.github.com/features/mastering-markdown/ | Docs: https://help.github.com/articles/github-flavored-markdown/ | Docs: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet | FileExt: ``.Md`` | FileExt: ``.md`` Markdown is a :ref:`Lightweight markup language` which can be parsed and transformed to valid :ref:`HTML-`. * GitHub and BitBucket support Markdown in Issue Descriptions, Wiki Pages, and Comments * :ref:`Jupyter Notebook` supports Markdown in Markdown cells .. index:: CommonMark .. _commonmark: CommonMark +++++++++++++ | Homepage: https://commonmark.org | Standard: https://spec.commonmark.org/0.29/ | Src: https://github.com/commonmark/commonmark-spec :ref:`CommonMark` is one effort to standardize :ref:`Markdown`. .. index:: MyST Markdown .. index:: MyST .. _myst: .. _myst markdown: MyST Markdown ++++++++++++++ | Src: https://github.com/executablebooks/MyST-Parser | Src: https://github.com/executablebooks/MyST-NB | Docs: https://myst-parser.readthedocs.io/en/latest/ | Docs: https://myst-nb.readthedocs.io/en/latest/ MyST Markdown is :ref:`CommonMark` :ref:`Markdown` with support for :ref:`Sphinx` roles and directives. - :ref:`jupyter-book` implemented MyST Markdown. .. index:: MediaWiki Markup .. _mediawiki markup: MediaWiki Markup ++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Help:Wiki_markup | Standard: https://www.mediawiki.org/wiki/Markup_spec | Docs: https://en.wikipedia.org/wiki/Help:Wiki_markup#Link_to_another_namespace | Docs: https://www.mediawiki.org/wiki/Help:Formatting | Docs: https://meta.wikimedia.org/wiki/Help:Wikitext_examples | Docs: https://en.wikipedia.org/wiki/Help:Displaying_a_formula MediaWiki Markup is a :ref:`Lightweight markup language` "WikiText" which can be parsed and transformed to valid :ref:`HTML-`. * Wikipedia is built on MediaWiki, which supports MediaWiki Markup. .. index:: RD .. index:: Ruby Document Format .. _rd: RD ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Ruby_Document_format | Standard: https://github.com/uwabami/rdtool/blob/master/doc/rd-draft.rd | Standard: https://github.com/uwabami/rdtool/blob/master/doc/rd-draft.rd.ja RD is a :ref:`Lightweight markup language` for documenting :ref:`Ruby` code and programs. .. index:: Rdoc .. _rdoc: RDoc ++++++ | Src: https://github.com/rdoc/rdoc | Docs: http://docs.seattlerb.org/rdoc/ | Docs: https://raw.githubusercontent.com/rdoc/rdoc/master/ExampleRDoc.rdoc RDoc is a tool and a :ref:`Lightweight markup language` for generating :ref:`HTML-` and command-line documentation for :ref:`Ruby` projects. To not build RDoc docs when installing a :ref:`Gem `: .. code:: bash gem install --no-rdoc --no-ri gem install --no-document gem install -N .. index:: ReStructuredText .. _restructuredtext: ReStructuredText ++++++++++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/ReStructuredText | Homepage: http://docutils.sourceforge.net/rst.html | Docs: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html | Docs: http://docutils.sourceforge.net/docs/ref/rst/directives.html | Docs: http://docutils.sourceforge.net/docs/ref/rst/roles.html | Docs: https://www.sphinx-doc.org/rest.html ReStructuredText (*ReST*, *RST*) is a :ref:`Lightweight markup language` commonly used for narrative documentation and inline Python, C, Java, etc. docstrings which can be parsed, transformed, and published to valid :ref:`HTML-`, ePub, LaTeX, PDF. :ref:`Sphinx` is built on :ref:`Docutils`, the primary implementation of ReStructuredText. :ref:`Pandoc` also supports a form of ReStructuredText. .. glossary:: ReStructuredText Directive Actionable blocks of ReStructuredText | Docs: http://docutils.sourceforge.net/docs/ref/rst/directives.html ``include``, ``contents``, and ``index`` are all ReStructuredDirectives: .. code-block:: rest .. include:: goals.rst .. contents:: Table of Contents :depth: 3 .. index:: Example 1 .. index:: Sphinx + .. _example-1: Sphinx +1 ========== This refs :ref:`example 1 `. Similarly, an explicit link to this anchor `<#example-1>`__ And an explicit link to this section `<#sphinx-1>`__ (which is otherwise not found in the source text). .. index:: Example 2 .. _example 2: Example 2 ========== This links to :ref:`example-1` and :ref:`example 2`. (`<#example-1>`__, `<#example-2>`__) And this also links to `Example 2`_. .. include:: LICENSE .. note:: ``index`` is a :ref:`Sphinx` Directive, which will print an error to the console when building but will otherwise silently dropped by non-Sphinx ReStructuredText parsers like :ref:`Docutils` (GitHub) and :ref:`Pandoc`. ReStructuredText Role RestructuredText role extensions | Docs: http://docutils.sourceforge.net/docs/ref/rst/roles.html ``:ref:`` is a :ref:`Sphinx` RestructuredText Role: .. code-block:: rest A (between files) link to :ref:`example 2`. .. index:: C .. _c: C ~~ | Wikipedia: ``__ | Docs: https://www.securecoding.cert.org/confluence/display/c/SEI+CERT+C+Coding+Standard | Docs: https://cwe.mitre.org/top25/#CWE-120 | Docs: https://cwe.mitre.org/data/definitions/120.html#Demonstrative_Examples | Docs: https://learnxinyminutes.com/docs/c/ C is a third-generation programming language which affords relatively low-level machine access while providing helpful abstractions. Every :ref:`Windows` kernel is written in C. The GNU/:ref:`Linux` kernel is written in C and often compiled by :ref:`GCC` or :ref:`Clang` for a particular architecture (see: ``man uname``) The :ref:`OSX` kernel is written in C. :ref:`Libc` libraries are written in C. Almost all of the projects linked here, at some point, utilize code written in C. .. index:: Libc .. _libc: Libc ++++++ | Wikipedia: https://en.wikipedia.org/wiki/C_POSIX_library A libc is a standard library of :ref:`C` routines. Libc implementations: * :ref:`Glibc` * :ref:`BSD Libc ` * https://en.wikipedia.org/wiki/UClibc * :ref:`Bionic` .. index:: GNU Libc .. index:: Glibc .. _glibc: ------ Glibc ------ | Wikipedia: https://en.wikipedia.org/wiki/GNU_C_Library | Homepage: https://www.gnu.org/software/libc/ | Docs: https://www.gnu.org/software/libc/documentation.html | Docs: https://www.gnu.org/software/libc/manual/html_mono/libc.html | Docs: https://sourceware.org/glibc/wiki/HomePage | Src: https://en.wikipedia.org/wiki/GNU_C_Library Glibc is the GNU :ref:`C` Library (:ref:`libc`). Many :ref:`Linux` packages and the :ref:`GNU/Linux ` kernel build from Glibc. .. index:: BSD Libc .. _bsd-libc: --------- BSD Libc --------- | Wikipedia: https://en.wikipedia.org/wiki/C_standard_library#BSD_libc | Src: https://svnweb.freebsd.org/base/head/lib/libc/ | Src: https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/ | Src: https://opensource.apple.com/source/Libc/ BSD libc are a superset of :ref:`POSIX`. :ref:`OSX` builds from BSD libc. :ref:`Android` :ref:`Bionic` is a BSD libc. .. index:: Bionic .. _bionic: ------- Bionic ------- | Wikipedia: ``__ | Src: git https://github.com/android/platform_bionic | Docs: https://developer.android.com/tools/sdk/ndk/index.html Bionic is the :ref:`Android` :ref:`libc`, which is a :ref:`BSD Libc `. .. index:: C++ .. _c++: C++ ~~~~ | Wikipedia: ``__ | Docs: https://learnxinyminutes.com/docs/c++/ C++ is a free and open source third-generation programming language which adds object orientation and a standard library to :ref:`C`. * C++ is an ISO specification: C++98, C++03, C++11 (C++0x), C++14, [ C++17 ] * There are many template libraries for C++: https://en.wikipedia.org/wiki/List_of_C%2B%2B_template_libraries .. index:: Standard Template Library .. _standard template library: Standard Template Library ++++++++++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Standard_Template_Library .. index:: libstdc++ .. _libstdc++: ----------- libstdc++ ----------- | Src: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libstdc%2B%2B-v3 | Src: https://github.com/gcc-mirror/gcc/tree/master/libstdc%2B%2B-v3 | Docs: https://gcc.gnu.org/onlinedocs/libstdc++/ libstdc++ is the free and open source GNU :ref:`C++` :ref:`Standard Template Library`. - :ref:`GCC` (:ref:`G++`) typically builds with libstdc++. ------------- libc++ ------------- | Homepage: https://libcxx.llvm.org/ | Src: https://github.com/llvm/llvm-project/tree/master/libcxx/ | Docs: https://libcxx.llvm.org/docs/ libc++ (libcxx) is the free and open source :ref:`LLVM` :ref:`C++` :ref:`Standard Template Library`. - :ref:`Clang` (:ref:`clang++`) typically builds with libc++ (libcxx). .. index:: Microsoft STL .. _microsoft stl: --------------- Microsoft STL --------------- | Src: https://github.com/microsoft/STL Microsoft STL is Microsoft's free and open source implementation of the :ref:`C++` :ref:`Standard Template Library`. * Microsoft Visual C++ typically builds with the Microsoft STL. .. index:: Boost .. _boost: Boost +++++++ | Wikipedia: ``__ | Homepage: https://www.boost.org/ | Src: https://github.com/boostorg/boost | Docs: https://www.boost.org/doc/ | Docs: https://www.boost.org/doc/libs/release/ | Docs: https://www.boost.org/doc/libs/release/more/getting_started/ | Docs: https://www.boost.org/doc/libs/release/libs/python/doc/html/ Boost is a free and open source set of :ref:`C++` libraries for doing lots of things in C++. .. index:: Fortran .. _fortran: Fortran ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Fortran Fortran (or FORTRAN) is a third-generation programming language frequently used for mathematical and scientific computing. Some of the :ref:`SciPy` libraries build optimized mathematical Fortran routines. .. index:: Haskell .. _haskell: Haskell ~~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.haskell.org/ | Download: https://www.haskell.org/downloads | Download: https://www.haskell.org/platform/ | Docs: https://www.haskell.org/documentation | Docs: https://learnxinyminutes.com/docs/haskell/ | Docs: http://learnyouahaskell.com/chapters | Docs: https://en.wikipedia.org/wiki/Haskell_features Haskell is a free and open source strongly statically typed purely functional programming language. :ref:`Cabal` is the Haskell package manager. :ref:`Pandoc` is written in Haskell. .. index:: Go .. _go: Go ~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://golang.org/ | Docs: https://golang.org/doc/ | Src: https://github.com/golang/go | Docs: https://learnxinyminutes.com/docs/go/ Go is a free and open source statically-typed :reF:`C`-based third generation language. .. index:: Java .. _Java: Java ~~~~~ | Wikipedia: ``__ | Docs: http://javadocs.org/ | Docs: https://learnxinyminutes.com/docs/java/ Java is a third-generation programming language which is compiled into code that runs in a virtual machine (``JVM``) written in :ref:`C` for many different operating systems. .. index:: JVM .. _jvm: JVM +++++ | Wikipedia: https://en.wikipedia.org/wiki/Java_virtual_machine A JVM ("Java Virtual Machine") runs :ref:`Java` code (classes and JARs). * There are JVMs available for very many platforms * Both the JRE and the JDK include a compiled JVM: + JRE -- Java Runtime Environment (End Users) + JDK -- Java Developer Kit (Developers) * Java SE is an implementation specification with things like ``java.lang`` and ``java.io`` and ``java.net`` * There are now multiple Java SE Implementations: | Wikipedia: https://en.wikipedia.org/wiki/Java_Platform,_Standard_Edition | https://en.wikipedia.org/wiki/Java_(software_platform)#History + Oracle Java (was **Sun Java**) | Wikipedia: https://en.wikipedia.org/wiki/Java_Development_Kit | Download: https://www.oracle.com/technetwork/java/javase/ | Download: https://www.oracle.com/technetwork/java/javase/downloads/ | Download: https://www.java.com/en/download/ | Docs: https://www.java.com/en/download/help/index_installing.xml?os=All+Platforms + OpenJDK (open source) | Wikipedia: https://en.wikipedia.org/wiki/OpenJDK | Homepage: https://openjdk.java.net/ | Download: https://openjdk.java.net/install/ | Src: https://hg.openjdk.java.net/ | Docs: https://wiki.openjdk.java.net/ | Docs: https://openjdk.java.net/guide/ + IcedTea (open source) | Wikipedia: https://en.wikipedia.org/wiki/IcedTea * Java EE ("Java Enterprise Edition") extends Java SE with a number of APIs for web services (``javax.servlet``, ``javax.transaction``) https://en.wikipedia.org/wiki/Java_Platform,_Enterprise_Edition .. index:: Javascript .. _js: .. _javascript: Javascript ~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/JavaScript | Docs: https://learnxinyminutes.com/docs/javascript/ Javascript (JS) is a free and open source third-generation programming language designed to run in an interpreter; now specified as :ref:`ECMAScript`. All major web browsers support Javascript. Client-side (web) applications can be written in Javascript. Server-side (web) applications can be written in Javascript, often with :ref:`Node.js`, :ref:`NPM`, and :ref:`Bower` packages. .. note:: Java and JavaScript are two distinctly different languages and developer ecosystems. .. index:: ECMAScript .. _ecmascript: ECMAScript +++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/ECMAScript | Homepage: https://www.ecma-international.org/ecma-262/ | Src: https://github.com/tc39/ecma262#ecmascript | Spec: https://www.ecma-international.org/ecma-262/ | Spec: https://tc39.github.io/ecma262/ | Spec: https://tc39.es/ecma262/ ECMAScript (ES) is an evolving, formally-specified, weakly-typed scripting language from which :ref:`Javascript` and ActionScript are derived. - There are multiple versions of ECMAScript (ES): - ES1 -- ES1997 - ES2 -- ES1998 - ES3 -- ES1999 - ES5 -- ES2009 - ES6 -- ES2015 - ES7 -- ES2016 - ES8 -- ES2017 - ES9 -- ES2018 - ES10 -- ES2019 - ES.Next - :ref:`Babel` compiles ECMAScript (ES6+) to Javascript. - Some browsers support various versions (ES7) of ECMAScript. - https://en.wikipedia.org/wiki/List_of_ECMAScript_engines - :ref:`Firefox` is built upon the SpiderMonkey ECMAScript engine. - Google :ref:`Chrome`, :ref:`Node.JS`, and the latest Microsoft :ref:`Edge` are built upon the V8 ECMAEscript engine. .. index:: Babel .. _babel: Babel ++++++ | Wikipedia: ``__ | Homepage: https://babeljs.io/ | Src: https://github.com/babel/babel | Docs: https://babeljs.io/docs/en/ Babel is a :ref:`Javascript` (:ref:`ECMAScript`) compiler that transforms ES6 (ES2015) and beyond into browser-compatible JS. - :ref:`ReactJS` developers commonly compile ES6+ and :ref:`JSX` to JS with Babel. .. index:: Node.js .. _nodejs: .. _node.js: Node.js +++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Node.js | Homepage: https://nodejs.org/ | Src: https://github.com/joyent/node | Docs: https://nodejs.org/en/docs/ | Docs: https://nodejs.org/api/ Node.js is a free and open source framework for :ref:`Javascript` applications written in :ref:`C`, :ref:`C++`, and :ref:`Javascript`. .. index:: Jinja2 .. _jinja2: Jinja2 ~~~~~~~ | Wikipedia: ``__ | Homepage: https://palletsprojects.com/p/jinja/ | Src: https://github.com/pallets/jinja | Docs: https://jinja2.readthedocs.io/en/latest/ | Docs: https://jinja.palletsprojects.com/ Jinja2 is a free and open source templating engine written in :ref:`Python`. :ref:`Sphinx` and :ref:`Salt` are two projects that utilize Jinja2. .. index:: Perl .. _perl: Perl ~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Perl | Homepage: https://www.perl.org/ | Project: https://dev.perl.org/perl5/ | Docs: https://www.perl.org/docs.html | Src: git git://perl5.git.perl.org/perl.git Perl is a free and open source, dynamically typed, :ref:`C`-based third-generation programming language. Many of the :ref:`Debian` system management tools are or were originally written in Perl. .. index:: Python .. _python: Python ~~~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.python.org/ | Src: git https://github.com/python/cpython | Src: git https://github.com/python/peps | Docs: https://docs.python.org/2/ | Docs: https://docs.python.org/3/ | Docs: https://docs.python.org/devguide/ | Docs: https://docs.python.org/devguide/documenting.html | Docs: https://wiki.python.org/moin/PythonBooks | Docs: https://www.onlineprogrammingbooks.com/python/ | Docs: https://www.reddit.com/r/learnpython/wiki/index | Docs: https://www.reddit.com/r/learnpython/wiki/books | Docs: https://www.quora.com/Python-programming-language-1/What-are-the-best-books-courses-for-learning-Python | Docs: https://en.wikiversity.org/wiki/Python | Docs: https://www.class-central.com/search?q=python | Docs: https://learnxinyminutes.com/docs/python/ | Awesome: https://github.com/vinta/awesome-python Python is a free and open source dynamically-typed, :ref:`C`-based third-generation programming language. As a multi-paradigm language with support for functional and object-oriented code, Python is often utilized for system administration and scientific software development. * Many of the :ref:`RedHat` system management tools (e.g. the :ref:`Yum` and :ref:`dnf` package managers) are written in Python. * The Gentoo :ref:`Portage` package manager is written in Python. * The :ref:`Conda` package manager is written in Python. * :ref:`IPython`, :ref:`Pip`, :ref:`Conda`, :ref:`Sphinx`, :ref:`Docutils`, :ref:`Mercurial`, :ref:`OpenStack`, :ref:`Libcloud`, :ref:`Salt`, :ref:`Ansible`, :ref:`Tox`, :ref:`Virtualenv`, and :ref:`Virtualenvwrapper` are all written in Python. * :ref:`PyPI` is the Python community index for sharing open source :ref:`python packages`. :ref:`Pip` installs from PyPI. The Python community is generously supported by a number of sponsors and the Python Infrastructure Team: * https://www.python.org/psf/sponsorship/ * https://www.python.org/psf/members/#sponsor-members * https://infra.psf.io/ * https://github.com/python/psf-salt * https://packaging.python.org/ * https://github.com/pypa/packaging.python.org .. index:: CPython .. _cpython: CPython ++++++++ | Wikipedia: ``_ | Homepage: https://www.python.org/ | Docs: https://docs.python.org/2/ | Docs: https://docs.python.org/devguide/ | Docs: https://docs.python.org/devguide/documenting.html | Docs: https://learnxinyminutes.com/docs/python/ | Src: hg https://hg.python.org/cpython CPython is the reference :ref:`Python` language implementation written in :ref:`C`. * https://github.com/python/cpython/blob/master/Grammar/Grammar CPython can interface with other :ref:`C` libraries through a number of interfaces: * https://docs.python.org/2/c-api/ * https://cffi.readthedocs.io/en/latest/ * :ref:`Cython` .. index:: Cython .. _cython: Cython ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Cython | Homepage: https://cython.org/ | PyPI: https://pypi.python.org/pypi/Cython | Docs: https://docs.cython.org/ | Docs: https://docs.cython.org/src/userguide/language_basics.html Cython is a superset of :ref:`CPython` which adds static type definitions; making :ref:`CPython` code faster, in many cases. .. index:: NumPy .. _numpy: NumPy ++++++ | Wikipedia: https://en.wikipedia.org/wiki/NumPy | Homepage: https://www.numpy.org/ | Src: https://github.com/numpy/numpy | Docs: https://docs.scipy.org/doc/numpy/ NumPy is a library of array-based mathematical functions implemented in :ref:`C` and :ref:`Python`. * https://nbviewer.jupyter.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-2-Numpy.ipynb * https://scipy-lectures.github.io/intro/numpy/index.html * https://scipy-lectures.github.io/advanced/advanced_numpy/index.html NumPy and other languages: * https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html * https://github.com/ipython/ipython/wiki/Extensions-Index .. index:: SciPy .. _scipy: SciPy ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/SciPy | Homepage: https://scipy.org/ | Src: https://github.com/scipy/scipy | Docs: https://www.scipy.org/docs.html | Docs: https://docs.scipy.org/doc/scipy/reference/ | Docs: https://www.scipy.org/install.html SciPy is a set of science and engineering libraries for :ref:`Python`, primarily written in :ref:`C`. * https://nbviewer.jupyter.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-3-Scipy.ipynb * https://scipy-lectures.github.io/intro/scipy.html * The :ref:`SciPy Stack ` specification includes the SciPy package and its dependencies. .. index:: SciPy Stack .. _scipystack: SciPy Stack +++++++++++++ | Docs: https://www.scipy.org/stackspec.html | Docs: https://www.scipy.org/install.html Python Distributions * :ref:`Sage Math` (:ref:`SageMathCloud`) * Enthought Canopy * Python(x,y) * WinPython * Pyzo * :ref:`Anaconda` (:ref:`Conda`, :ref:`Wakari`) .. index:: Scipy Stack Docker Containers .. _scipy stack docker containers: SciPy Stack Docker Containers +++++++++++++++++++++++++++++++ .. index:: Jupyter Docker Stacks .. _jupyter docker stacks: ----------------------------- Jupyter Docker Stacks ----------------------------- | Src: https://github.com/jupyter/docker-stacks | DockerHub: https://hub.docker.com/r/ipython/ipython/ | DockerHub: https://hub.docker.com/r/jupyter/ | DockerHub: https://hub.docker.com/r/jupyter/datascience-notebook/ | DockerHub: https://hub.docker.com/r/jupyter/tmpnb/ * :ref:`Jupyter` and :ref:`Scipystack` :ref:`Docker` containers: * https://hub.docker.com/r/jupyter/datascience-notebook/ * https://github.com/jupyter/docker-stacks/tree/master/datascience-notebook#docker-options * :ref:`conda` * https://github.com/jupyter/tmpnb#using-jupyterdocker-stacks-images See also: + :ref:`Docker` + https://wrdrd.github.io/docs/consulting/data-science#reproducibility .. index:: PyPy .. _pypy: PyPy +++++ | Wikipedia: https://en.wikipedia.org/wiki/PyPy | Homepage: https://pypy.org/ | Src: https://bitbucket.org/pypy/pypy | Docs: http://buildbot.pypy.org/waterfall | Docs: https://pypy.readthedocs.io/en/latest/ | Docs: https://pypy.readthedocs.io/en/latest/introduction.html PyPy is a JIT LLVM compiler for :ref:`Python` code written in RPython -- a restricted subset of :ref:`CPython` syntax -- which compiles to :ref:`C`, and is often faster than :ref:`CPython` for many types of purposes. .. index:: NumPyPy .. _numpypy: NumPyPy ++++++++ NumPyPy is a port of :ref:`NumPy` to :ref:`PyPy`: | Src: https://bitbucket.org/pypy/numpypy | Docs: http://buildbot.pypy.org/numpy-status/latest.html | Docs: https://pypy.org/numpydonate.html .. index:: Python 3 .. _python3: Python 3 ++++++++++ | Docs: https://docs.python.org/3/ | Docs: https://docs.python.org/3/howto/pyporting.html | Docs: https://docs.python.org/3/howto/cporting.html | Docs: https://learnxinyminutes.com/docs/python3/ Python 3 made a number of incompatible changes, requiring developers to update and review their Python 2 code in order to "port to" Python 3. Python 2 will be supported in "no-new-features" status for quite some time. Python 3 Wall of Superpowers tracks which popular packages have been ported to support Python 3: https://python3wos.appspot.com/ There are a number of projects which help bridge the gap between the two language versions: * https://pypi.python.org/pypi/six * https://six.readthedocs.io/ * https://pypi.python.org/pypi/nine * https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py * https://pypi.python.org/pypi/future * https://python-future.org/ See also: :ref:`Anaconda` .. index:: awesome-python-testing .. _awesome-python-testing: awesome-python-testing ++++++++++++++++++++++++ | Homepage: https://westurner.github.io/wiki/awesome-python-testing.html | Src: https://github.com/westurner/wiki/blob/master/awesome-python-testing.rest .. index:: Tox .. _tox: Tox ++++++++++++++ | Homepage: https://testrun.org/tox/ | Docs: https://tox.readthedocs.io/en/latest/ | Src: https://github.com/tox-dev/tox | Pypi: https://pypi.python.org/pypi/tox Tox is a build automation tool designed to build and test Python projects with multiple language versions and environments in separate :ref:`virtualenvs `. Run the py27 environment:: tox -v -e py27 tox --help .. index:: Ruby .. _ruby: Ruby ~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.ruby-lang.org/ | Src: https://github.com/ruby/ruby | Docs: https://www.ruby-lang.org/en/documentation/ | Docs: https://learnxinyminutes.com/docs/ruby/ Ruby is a free and open source dynamically-typed programming language. :ref:`Vagrant` is written in Ruby. .. index:: Rust .. _rust: Rust ~~~~~ | Wikipedia: ``__ | Homepage: https://www.rust-lang.org/ | Docs: https://doc.rust-lang.org/stable/ | Docs: https://doc.rust-lang.org/nightly/ | Docs: https://learnxinyminutes.com/docs/rust/ | Docs: https://doc.rust-lang.org/book/ Rust is a free and open source strongly typed multi-paradigm programming language. * It's possible to "drop-in replace" :ref:`C` and :ref:`C++` modules with rust code * Rust can call into :ref:`C` code * Rust is similar to but much safer than :ref:`C` and :ref:`C++` ("memory safety") * In terms of :ref:`C` compatibility and smart pointers/references ("memory safety"); :ref:`Go` and :ref:`Rust` have similar objectives. https://doc.rust-lang.org/book/ownership.html * https://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization * https://en.wikipedia.org/wiki/Smart_pointer * https://doc.rust-lang.org/book/the-stack-and-the-heap.html * https://rust-lang.github.io/rlibc/rlibc/fn.memcpy.html .. code:: pub unsafe extern fn memcpy(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 .. index:: Scala .. _scala: Scala ~~~~~~~ | Wikipedia: ``__ | Homepage: https://scala-lang.org/ | Src: git https://github.com/scala/scala | Twitter: https://twitter.com/scala_lang | Docs: https://scala-lang.org/api/current/ | Docs: https://scala-lang.org/api/current/#scala.collection.mutable.LinkedHashMap | Docs: https://learnxinyminutes.com/docs/scala/ Scala is a free and open source object-oriented and functional :ref:`programming language` which compiles to :ref:`JVM` (and :ref:`LLVM`) bytecode. .. index:: TypeScript .. _typescript: TypeScript ~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/TypeScript | Homepage: https://www.typescriptlang.org/ | Src: https://github.com/Microsoft/TypeScript | NPM: https://www.npmjs.com/package/typescript | NPMPkg: typescript | Standard: https://github.com/Microsoft/TypeScript/blob/master/doc/spec.md | FileExt: ``.ts`` | Docs: https://www.typescriptlang.org/Tutorial | Docs: https://www.typescriptlang.org/Handbook | Docs: https://learnxinyminutes.com/docs/typescript/ TypeScript is a free and open source :ref:`Programming Language` developed as a superset of :ref:`Javascript` with optional additional features like static typing and native object-oriented code. * Angular 2 is written in :ref:`TypeScript`: https://github.com/angular/angular/blob/master/modules/angular2/angular2.ts .. index:: WebAssembly .. _webassembly: WebAssembly ~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/WebAssembly | Project: https://www.w3.org/community/webassembly/ | Src: https://github.com/WebAssembly | FileExt: ``.wast`` | FileExt: ``.wasm`` | Docs: https://github.com/WebAssembly/design | Docs: https://github.com/WebAssembly/design/blob/master/UseCases.md WebAssembly (*wasm*) is a safe (sandboxed), efficient low-level :ref:`programming language` (abstract syntax tree) and binary format for the web. * WebAssembly is initially derived from asm.js and PNaCL. * WebAssembly is an industry-wide effort. * :ref:`LLVM` can generate WebAssembly from e.g. :ref:`C` and :ref:`C++` code. .. index:: YAML .. _yaml: YAML ~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/YAML | Homepage: https://yaml.org | Docs: https://learnxinyminutes.com/docs/yaml/ YAML ("YAML Ain't Markup Language") is a concise data serialization format. Most :ref:`Salt` states and pillar data are written in YAML. Here's an example ``top.sls`` file: .. code-block:: yaml base: '*': - openssh '*-webserver': - webserver '*-workstation': - gnome - i3 .. index:: Compilers .. _compilers: Compilers ========== | Wikipedia: https://en.wikipedia.org/wiki/Compiler | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Compilers * History of compiler construction https://en.wikipedia.org/wiki/History_of_compiler_construction .. index:: Interpreters .. _interpreters: Interpreter ~~~~~~~~~~~~~ | Wikipedia: ``__ | WikipediaCategory: ``__ * https://en.wikipedia.org/wiki/List_of_command-line_interpreters * https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Operating_system_shells * https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Programming * https://en.wikipedia.org/wiki/List_of_command-line_interpreters#Programming_languages * A programming :ref:`language` interpreter interprets instructions without an ahead-of-time compilation step. * A programming language interpreter is generally compiled ahead-of-time. * A programming language interpreter implements a version of a programming language specification. * Command :ref:`Shells` (e.g. :ref:`Bash`, :ref:`ZSH`) are programming language interpreters. * Scripting :ref:`Languages` (e.g. :ref:`PERL`, :ref:`Ruby`, :ref:`Python`) are interpreted or interpret-able languages. * Interpreted languages can often also be compiled ahead-of-time (e.g. with build optimizations). .. index:: Source-to-Source Compiler .. _source-to-source compiler: Source-to-Source compiler ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Source-to-source_compiler | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Source-to-source_compilers .. contents: :local: .. index:: ROSE .. _rose: ROSE ++++++ | Wikipedia: ``__ | Homepage: http://rosecompiler.org/ | Src: git https://github.com/rose-compiler/rose | Docs: http://rosecompiler.org/?page_id=11 | Docs: http://rosecompiler.org/ROSE_Tutorial/ROSE-Tutorial.pdf | Docs: http://www.rosecompiler.org/ROSE_HTML_Reference/ | Docs: http://wiki.rosecompiler.org/ * http://rosecompiler.org/ROSE_Tutorial/ROSE-Tutorial.pdf .. index:: Binutils .. index:: GNU Binutils .. _binutils: Binutils ~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNU_Binutils | Homepage: https://www.gnu.org/software/binutils/ | Src: git git://sourceware.org/git/binutils-gdb.git | Docs: https://sourceware.org/binutils/docs-2.24/ | Docs: https://sourceware.org/binutils/docs-2.24/binutils/index.html | Docs: https://sourceware.org/binutils/docs-2.24/as/index.html | Docs: https://sourceware.org/binutils/docs-2.24/ld/index.html GNU Binutils are a set of utilities for working with assembly and binary. :ref:`GCC` utilizes GNU Binutils to compile the GNU/:ref:`Linux` kernel and userspace. GAS, the GNU Assembler (``as``) assembles ASM code for linking by the GNU linker (``ld``). .. index:: Clang .. _clang: Clang ~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Clang | Homepage: https://clang.llvm.org/ | Src: git https://github.com/llvm/llvm-project | Docs: https://clang.llvm.org/docs/ | Docs: https://clang.llvm.org/docs/UsersManual.html Clang is a compiler front end for :ref:`C`, :ref:`C++`, and Objective C/++. Clang is part of the :ref:`LLVM` project. .. index:: GCC .. index:: GNU Compiler Collection .. _gcc: GCC ~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNU_Compiler_Collection | Homepage: https://gcc.gnu.org/ | Docs: https://gcc.gnu.org/onlinedocs/ | Src: git ``__ The GNU Compiler Collection started as a Free and Open Source compiler for :ref:`C`. - There are now GCC frontends for many languages, including :ref:`C++`, :ref:`Fortran`, :ref:`Java`, and :ref:`Go`. - The :ref:`C++` GCC frontend binary is called ``g++``. .. index:: GNU Linker .. index:: ld .. _ld: GNU Linker ~~~~~~~~~~~~ | Wikipedia: ``__ The GNU Linker is the GNU implementation of the ``ld`` command for linking object files and libraries. .. index:: LLVM .. _llvm: LLVM ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/LLVM | Homepage: https://llvm.org/ | Src: git https://github.com/llvm/llvm-project | Docs: https://llvm.org/docs/ | Docs: https://llvm.org/docs/GettingStarted.html | Docs: https://llvm.org/docs/ReleaseNotes.html | Docs: https://llvm.org/ProjectsWithLLVM/ LLVM "*Low Level Virtual Machine*" is a reusable compiler infrastructure with frontends for many languages. * :ref:`Clang` is an LLVM frontend for C-based languages like :ref:`C`, :ref:`C++`, :ref:`CUDA`, and :ref:`OpenCL`. - There is a :ref:`WASM` LLVM backend: LLVM can produce :ref:`WebAssembly` binaries. - The :ref:`C++` LLVM frontend binary is called ``clang++``. .. index:: Operating Systems .. _operating systems: Operating Systems =================== | Wikipedia: https://en.wikipedia.org/wiki/Operating_system | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Operating_systems | WikipediaCategory: https://en.wikipedia.org/wiki/Category:Operating_system_technology * History of operating systems https://en.wikipedia.org/wiki/History_of_operating_systems * Timeline of operating systems https://en.wikipedia.org/wiki/Timeline_of_operating_systems * List of operating systems https://en.wikipedia.org/wiki/List_of_operating_systems * Comparison of operating systems https://en.wikipedia.org/wiki/Comparison_of_operating_systems * https://distrowatch.com/ * https://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems .. index:: POSIX .. _posix: POSIX ~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/POSIX | Docs: https://en.wikipedia.org/wiki/POSIX#POSIX-oriented_operating_systems POSIX ("Portable Operating System Interface") is a set of standards for :ref:`Shells`, :ref:`Operating Systems`, and APIs. .. index:: GNU/Linux .. index:: Linux .. _linux: Linux ~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Linux | Homepage: https://www.kernel.org/ | Docs: https://www.kernel.org/doc/ | Src: git https://github.com/torvalds/linux GNU/Linux ("Linux") is a free and open source operating system kernel written in :ref:`C`. .. code-block:: bash uname -a; echo "Linux" uname -o; echo "GNU/Linux" .. index:: Linux Distributions .. _linux-distributions: Linux Distributions ~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Linux_distribution A *Linux Distribution* is a collection of :ref:`Packages` compiled to work with a :ref:`GNU/Linux ` kernel and a :ref:`libc`. * https://commons.wikimedia.org/wiki/File:Linux_Distribution_Timeline_with_Android.svg .. index:: Arch .. _arch: Arch ++++++ | Wikipedia: ``__ | Homepage: https://www.archlinux.org/ | Download: https://www.archlinux.org/download/ | Docs: https://wiki.archlinux.org/ | Docs: https://aur.archlinux.org/ | Docs: https://aur4.archlinux.org/ | Docs: https://wiki.archlinux.org/index.php/Arch_packaging_standards | Docs: https://wiki.archlinux.org/index.php/Arch_User_Repository#AUR_4 Arch Linux is a :ref:`Linux Distribution ` that is built from :ref:`AUR` packages. .. index:: Debian .. _debian: Debian +++++++++++++++++ | Wikipedia: ``__ | Homepage: https://www.debian.org/ | Download: https://www.debian.org/distrib/ | DockerHub: https://hub.docker.com/_/debian/ | Docs: https://www.debian.org/doc/ | Docs: https://www.debian.org/doc/manuals/debian-reference/ | Docs: https://www.debian.org/doc/#manuals | Docs: https://www.debian.org/doc/debian-policy/ (main, contrib, non-free) | Docs: https://www.debian.org/releases/stable/releasenotes | Docs: https://www.debian.org/releases/stable/i386/release-notes/ | Docs: https://www.debian.org/releases/stable/amd64/release-notes/ Debian is a :ref:`Linux Distribution ` that is built from :ref:`DEB` packages. .. index:: Ubuntu .. _ubuntu: Ubuntu +++++++++++++++++ | Wikipedia: ``_ | Homepage: https://ubuntu.com/ | Src: https://launchpad.net/ubuntu | Src: http://archive.ubuntu.com/ | Src: http://releases.ubuntu.com/ | Download: https://ubuntu.com/download | DockerHub: https://hub.docker.com/_/ubuntu/ | Docs: https://help.ubuntu.com/ | Q&A: https://askubuntu.com Ubuntu is a :ref:`Linux Distribution ` that is built from :ref:`DEB` packages which are often derived from :ref:`Debian` packages. .. index:: Fedora .. _fedora: Fedora +++++++ | Wikipedia: ``__ | Homepage: https://getfedora.org/ | Download: https://getfedora.org/en/workstation/download/ | Download: https://getfedora.org/en/server/download/ | Download: https://getfedora.org/en/cloud/download/ | DockerHub: https://hub.docker.com/_/fedora/ | Docs: https://docs.fedoraproject.org/en-US/index.html | Docs: https://fedoraproject.org/wiki/Fedora_Project_Wiki | Docs: https://fedoraproject.org/wiki/EPEL Fedora is a :ref:`Linux Distribution ` that is built from :ref:`RPM` packages. .. index:: RedHat .. index:: RedHat Enterprise Linux .. index:: RHEL .. _redhat: RedHat ++++++++ | Wikipedia: ``__ | Homepage: https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux | Docs: https://access.redhat.com/documentation/en-US/ | Docs: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/ RedHat Enterprise Linux ("RHEL") is a :ref:`Linux Distribution ` that is built from :ref:`RPM` packages. .. index:: CentOS .. _centos: -------- CentOS -------- | Wikipedia: https://en.wikipedia.org/wiki/CentOS | Homepage: https://www.centos.org/ | Download: https://www.centos.org/download/ | Docs: https://wiki.centos.org/ | Docs: https://www.centos.org/docs/ | DockerHub: https://hub.docker.com/_/centos/ CentOS is a :ref:`Linux Distribution ` that is built from :ref:`RPM` packages which is derived from :ref:`RHEL `. .. index:: Scientific Linux .. _scientific-linux: Scientific Linux ----------------- | Wikipedia: https://en.wikipedia.org/wiki/Scientific_Linux | Homepage: https://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux is a :ref:`Linux Distribution ` that is built from :ref:`RPM` packages which is derived from :ref:`CentOS`. which is derived from :ref:`RHEL `. * ``rdfs:seeAlso`` :ref:`Anaconda` (:ref:`Conda`) * ``rdfs:seeAlso`` :ref:`Portage` .. index:: Oracle Linux .. _oracle-linux: ------------ Oracle ------------ | Wikipedia: https://en.wikipedia.org/wiki/Oracle_Linux | Homepage: https://www.oracle.com/linux Oracle Linux is a :ref:`Linux Distribution ` that is built from :ref:`RPM` packages which is derived from :ref:`RHEL `. .. index:: Gentoo .. _gentoo: Gentoo ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Gentoo | Homepage: https://gentoo.org/ | Src: https://github.com/gentoo | Src: git https://github.com/gentoo/portage | Docs: https://wiki.gentoo.org/wiki/ | Docs: https://wiki.gentoo.org/wiki/Handbook:Main_Page | Docs: https://wiki.gentoo.org/wiki/Handbook:AMD64 | Docs: https://wiki.gentoo.org/wiki/Handbook:X86 | Docs: https://wiki.gentoo.org/wiki/Project:Portage | Docs: https://wiki.gentoo.org/wiki/Project:Hardened Gentoo is a :ref:`Linux Distribution ` built on :ref:`Portage`. * https://hub.docker.com/search?q=gentoo (Stage 3 + Portage) .. index:: ChromiumOS .. _chromiumos: ChromiumOS +++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Chromium_OS | Homepage: https://www.chromium.org/chromium-os | Docs: https://www.chromium.org/chromium-os/quick-start-guide | Docs: https://www.chromium.org/chromium-os/developer-guide | Src: https://chromium.googlesource.com/ (``chromiumos*/``) ChromiumOS is a :ref:`Linux Distribution ` built on :ref:`Portage`. .. index:: Crouton .. _crouton: ------------ Crouton ------------ | Src: https://github.com/dnschneid/crouton Crouton ("Chromium OS Universal Chroot Environment") installs and debootstraps a :ref:`Linux Distribution ` (i.e. :ref:`Debian` or :ref:`Ubuntu`) within a :ref:`ChromiumOS` or :ref:`ChromeOS` chroot. .. index:: ChromeOS .. _chromeos: ChromeOS +++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Chrome_OS ChromeOS is a :ref:`Linux Distribution ` built on :ref:`ChromiumOS` and :ref:`Portage`. * ChromeOS powers Chromebooks * https://en.wikipedia.org/wiki/Chromebook * ChromeOS powers Chromeboxes * https://en.wikipedia.org/wiki/Chromebox .. index:: CoreOS .. _coreos: CoreOS ++++++++ | Wikipedia: https://en.wikipedia.org/wiki/CoreOS | Homepage: https://coreos.com/ | Src: https://github.com/coreos | Docs: https://coreos.com/docs/ | Docs: https://coreos.com/docs/#running-coreos | Docs: https://coreos.com/docs/running-coreos/platforms/vagrant/ | Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos.md CoreOS is a :ref:`Linux Distribution ` for highly available distributed computing. CoreOS schedules redundant :ref:`docker` images with **fleet** and **systemd** according to configuration stored in **etcd**, a key-value store with a D-Bus interface. * CoreOS runs on very many platforms * CoreOS does not provide a package manager * CoreOS schedules Docker * CoreOS -- Operating System * etcd -- Consensus and Discovery * rkt -- Container Runtime * fleet -- Distributed init system (etcd, systemd) * flannel -- Networking .. index:: SteamOS .. _steamos: SteamOS ++++++++ | Wikipedia: | Homepage: https://store.steampowered.com/steamos | Download: https://store.steampowered.com/steamos/download | Docs: https://store.steampowered.com/steamos/oem | Docs: https://store.steampowered.com/steamos/buildyourown | DistroWatch: https://distrowatch.com/table.php?distribution=steamos SteamOS is a :ref:`Linux Distribution ` for gaming based on :ref:`Debian`. * SteamOS uses :ref:`Apt`, :ref:`Dpkg` and :ref:`DEB` :ref:`Packages` * Steam Machines run SteamOS * SteamOS runs Steam: https://en.wikipedia.org/wiki/Steam * https://en.wikipedia.org/wiki/Kerbal_Space_Program .. index:: Linux Notes .. _linux-notes: Linux Notes +++++++++++++ * https://github.com/westurner/provis * https://github.com/saltstack/salt-bootstrap curl -L https://bootstrap.saltstack.com scripts/bootstrap-salt.sh * Masterless Salt Config: ``make salt_local_highstate_test`` * [ ] Workstation role ---------------- Linux Dual Boot ---------------- * [ ] GRUB chainloader to partition boot record * Ubuntu and Fedora GRUB try to autodiscover Windows partitions .. index:: Android .. _android: Android +++++++++ | Wikipedia: ``__ | Homepage: https://www.android.com/ | Homepage: https://developer.android.com/ .. index:: Android SDK .. _android sdk: ------------- Android SDK ------------- | Homepage: https://developer.android.com/sdk/ | Src: https://android.googlesource.com/ | Src: https://github.com/android | Docs: https://developer.android.com/sdk/ | Docs: https://developer.android.com/sdk/installing/index.html | Docs: https://developer.android.com/sdk/installing/adding-packages.html | Docs: https://source.android.com/source/index.html | Docs: https://source.android.com/source/downloading.html | Docs: https://source.android.com/source/developing.html | Docs: https://source.android.com/source/contributing.html | Docs: https://sites.google.com/a/android.com/tools/build | Docs: https://developer.android.com/tools/workflow/index.html .. index:: Android Studio .. _android studio: ---------------- Android Studio ---------------- | | Homepage: https://developer.android.com/tools/studio/index.html | Docs: https://developer.android.com/tools/workflow/index.html | Docs: https://sites.google.com/a/android.com/tools/build/studio .. index:: Apple OSX .. index:: OS X .. index:: OSX .. _osx: OS X ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/OS_X | Wikipedia: https://en.wikipedia.org/wiki/MacOS | Homepage: https://www.apple.com/osx | Homepage: https://www.apple.com/macos/ | Docs: https://developer.apple.com/technologies/mac/ | Docs: https://developer.apple.com/library/mac/navigation/ | Docs: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/ | Src: https://www.apple.com/opensource/ OS X is a UNIX operating system based upon the Mach kernel from NeXTSTEP, which was partially derived from NetBSD and FreeBSD. OS X GUI support is built from XFree86/X.org :ref:`X11`. OS X maintains forks of many POSIX BSD and GNU tools like ``bash``, ``readlink``, and ``find``. :ref:`Homebrew` installs and maintains packages for OS X. .. code-block:: bash uname; echo "Darwin" .. index:: iOS .. _iOS: iOS +++++ | Wikipedia: https://en.wikipedia.org/wiki/IOS | Homepage: https://www.apple.com/ios/ iOS is a closed source UNIX operating system based upon many components of :ref:`OSX` adapted for phones and then tablets. * iOS powers iPhones and iPads * You must have a Mac with :ref:`OSX` and XCode to develop and compile for iOS. OSX Notes ++++++++++ * [ ] Create a fresh install :ref:`OSX` USB drive (16GB+) * http://osxdaily.com/2014/10/16/make-os-x-yosemite-boot-install-drive/ * Docs: https://support.apple.com/en-us/HT201372 * https://github.com/westurner/dotfiles/blob/master/scripts/ ``setup_*.sh`` * [ ] Manually update to latest versions (of zip, tar.gz, .dmg) * [ ] Port / wrap :ref:`shell ` scripts to / with :term:`salt formulas` and parameters (per-subnet, group, machine, os; :term:`salt pillar`): + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_brew.sh # :ref:`homebrew` + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_mavericks_python.sh # :ref:`python` + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_chrome.sh # :ref:`chrome` + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_chromium.sh # :ref:`chrome` + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_firefox.sh # :ref:`firefox` + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_adobereader.sh # PDF forms, signatures, annotations + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_vlc.sh (`vlc`) + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_f.lux.sh (`f.lux`, `UBY`) + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_powerline_fonts.sh (`UBY`) + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_macvim.sh (:ref:`vim`) + [ ] https://github.com/westurner/dotfiles/blob/master/scripts/setup_miniconda.sh (:ref:`conda`) * https://github.com/westurner/provis * [ ] https://github.com/westurner/provis/compare/feature/osx_support * [ ] create / remap "root" group * [ ] http://docs.saltstack.com/en/latest/topics/installation/osx.html ``brew install saltstack`` OR ``pip install salt`` -------------- OSX Reinstall -------------- * [ ] Generate installation media * [ ] Reboot to recovery partition * [ ] Adjust partitions * [ ] Format? * [ ] Install OS * [ ] (wait) * [ ] Manual time/date/language config * [ ] Run workstation provis scripts ------------------ OSX Fresh Install ------------------ * [ ] Generate / obtain installation media * [ ] Boot from installation media * [ ] Manual time/date/language config * [ ] Run workstation provis scripts -------------- OSX Dual Boot -------------- * https://www.howtogeek.com/187410/how-to-install-and-dual-boot-linux-on-a-mac/ * https://www.rodsbooks.com/refind/installing.html#osx .. index:: Windows .. _windows: Windows ~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Microsoft_Windows | Homepage: https://www.microsoft.com/en-us/windows/ | Docs: https://www.microsoft.com/enable/products/docs/ | Docs: Microsoft Windows is a NT-kernel based operating system. * There used to be a POSIX compatibility mode. * Chocolatey maintains a set of :ref:`NuGet` packages for Windows. .. index:: Windows Subsystem for Linux .. index:: WSL .. _wsl: .. _windows subsystem for linux: Windows Subsystem for Linux ++++++++++++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux | Homepage: https://docs.microsoft.com/en-us/windows/wsl | Source: https://github.com/Microsoft/WSL Windows Subsystem for Linux (*WSL*) is a binary compatibility layer which allows many Linux programs to be run on Windows 10+. The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a virtual machine. - Windows Subsystem for Linux is not a complete :ref:`virtualization` solution; but it does allow you to run e.g. :ref:`Ubuntu` or :ref:`Fedora` (and thus e.g. :ref:`Bash`) on a Windows machine. - :ref:`Docker` for Windows is one alternative to Windows Subsystem for Linux. .. index:: Windows Sysinternals .. _windows sysinternals: Windows Sysinternals +++++++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Sysinternals | Homepage: https://technet.microsoft.com/en-us/sysinternals | Download: https://technet.microsoft.com/en-us/sysinternals/bb842062 | Download: https://live.sysinternals.com/ | Docs: https://technet.microsoft.com/en-us/sysinternals/bb545027 | Docs: https://blogs.technet.microsoft.com/sysinternals/ Windows Sysinternals is a group of tools for working with :ref:`Windows`. * | Download: https://technet.microsoft.com/en-us/sysinternals/bb842062 * File & Disk: https://technet.microsoft.com/en-us/sysinternals/bb545046 * Networking: https://technet.microsoft.com/en-us/sysinternals/bb795532 * Process: https://technet.microsoft.com/en-us/sysinternals/bb795533 * Process Explorer: https://technet.microsoft.com/en-us/sysinternals/processexplorer * You can replace Task Manager (``taskman.exe``) with Process Explorer (``procexp.exe``) in many versions of Windows (e.g. so that ``ctrl-alt-delete`` launches ``procexp.exe`` * ``Win+R`` launches the run dialog .. code:: ## Win+R commands C:/ # open explorer to 'C:/' %UserProfile% # open the user profile env var directory cmd[.exe] # open a command prompt calc # calculator control # control panel services.msc # services management compmgmt.msc # computer management eventvwr.msc # event viewer Wupdmgr # windows update manager * https://www.pcsteps.com/1675-run-command-complete-list-windows/ * ``__ + ``%PATH%`` determines which commands you can run without typing a full absolute path. (Some programs will not run if other programs are not in a directory included in a user's current ``%PATH%`` variable) + The ``%PATH%`` variable is determined by key value pairs in the Windows Registry and can be set for: + The current prompt (with ``SET PATH=C:/additionalpath;%PATH%``) + A user + The machine https://stackoverflow.com/questions/8358265/how-to-update-path-variable-permanently-from-cmd-windows * Security: https://technet.microsoft.com/en-us/sysinternals/bb795534 * System Information: https://technet.microsoft.com/en-us/sysinternals/bb795535 * Miscellaneous: https://technet.microsoft.com/en-us/sysinternals/bb842059 .. index:: WSUS Offline Update .. _wsus offline update: WSUS Offline Update ++++++++++++++++++++++ | Homepage: http://www.wsusoffline.net/ | Download: http://download.wsusoffline.net/ | Src: svn https://svn.wsusoffline.net/svn/wsusoffline/trunk/ | Docs: http://www.wsusoffline.net/docs/ WSUS Offline Update is a free and open source software tool for generating offline :ref:`Windows` upgrade CDs / DVDs containing the latest upgrades for Windows, Office, and .Net. * Bandwidth costs: Windows Updates (WSUS) in GB * n_machines (see also: *Debtorrent*, :ref:`Packages`) * "Slipstreaming" an installation ISO is one alternative way to avoid having to spend hours upgrading a factory reinstalled ("reformatted") :ref:`Windows` installation Windows Notes +++++++++++++++ A few annotated excerpts from this Chocolatey :ref:`NuGet` :ref:`PowerShell` script https://gist.github.com/westurner/10950476#file-cinst_workstation_minimal-ps1 :: cinst GnuWin cinst sysinternals # Process Explorer XP cinst 7zip cinst curl * [ ] Install Chocolatey NuGet package manager: http://chocolatey.org * [ ] Install packages listed here: https://gist.github.com/westurner/10950476 * [ ] (Optional) uncomment salt first (optionally specify master) [OR Install salt] * [ ] Install salt: https://docs.saltstack.com/en/latest/topics/installation/windows.html * ``+R`` (Start > Run) * [ ] Run ``services.msc`` and log/prune unutilized services (e.g. workstation, server) and record changes made * https://en.wikipedia.org/wiki/Windows_service * https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.win_service.html * https://docs.saltstack.com/en/latest/ref/states/providers.html#provider-service ------------------ Windows Dual Boot ------------------ * [ ] Windows MBR chain loads to partition GRUB (`Linux`_) * [ ] Ubuntu WUBI .exe Linux Installer (XP, 7, 8*) * It's now better to install to a separate partition from a bootable ISO ----- UEFI ----- | Wikipedia: https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface * https://help.ubuntu.com/community/UEFI * Cygwin Windows Linux Userspace: ~ https://chocolatey.org/packages/Cygwin * https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c * http://winappdbg.sourceforge.net/#related-projects .. index:: Configuration Management .. _configuration management: Configuration Management ========================== | Wikipedia: https://en.wikipedia.org/wiki/Software_configuration_management | Wikipedia: https://en.wikipedia.org/wiki/Comparison_of_open-source_configuration_management_software .. index:: Ansible .. _ansible: Ansible ~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://ansible.com/ | Src: https://github.com/ansible/ansible | Docs: https://docs.ansible.com/ | Docs: https://docs.ansible.com/ansible/latest/ | Docs: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html | Docs: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html | Docs: https://docs.ansible.com/ansible/latest/reference_appendices/galaxy.html | Docs: https://github.com/ansible/molecule | Learnxinyminutes: https://learnxinyminutes.com/docs/ansible/ Ansible is a :ref:`Configuration Management` tool written in :ref:`Python` which runs idempotent Ansible Playbooks written in :ref:`YAML` markup with :ref:`Jinja2` variables for managing one or more physical and virtual machines running various operating systems over SSH. .. index:: Cobbler .. _cobbler: Cobbler ~~~~~~~~ | Wikipedia: ``__ | Homepage: https://cobbler.github.io/ | Download: https://cobbler.github.io/downloads/2.8.x.html | Src: git https://github.com/cobbler/cobbler | Docs: https://cobbler.github.io/manuals/quickstart/ | Docs: https://cobbler.github.io/manuals/2.8.0/ Cobbler is a machine image configuration, repository mirroring, and networked booting server with support for DNS, DHCP, TFTP, and PXE. * Cobbler can template kickstart files for the :ref:`RedHat` Anaconda installer * Cobbler can template :ref:`Debian` preseed files * Cobbler can PXE boot an ISO over TFTP (and unattended install) * BusyBox, :ref:`SystemRescueCD`, :ref:`Clonezilla` * Cobbler can manage a set of DNS and DHCP entries for physical systems * Cobbler can batch mirror :ref:`RPM` and :ref:`DEB` repositories (see also: `apt-cacher-ng`, :ref:`nginx`) * Cobbler-web is a Django WSGI application; usually configured with :ref:`Apache HTTPD` and mod_wsgi. * Cobbler-web delegates very many infrastructure privileges See also: `crowbar`, :ref:`OpenStack` Ironic bare-metal deployment .. index:: JuJu .. _juju: Juju ~~~~~~ | Wikipedia: ``__ | Homepage: https://jaas.ai/ | Src: https://github.com/juju/juju | Docs: https://jaas.ai/docs | Docs: https://jaas.ai/search?type=charm | TcpPort: 8001 Juju is a :ref:`Configuration Management` tool written in :ref:`Python` which runs Juju Charms written in :ref:`Python` on one or more systems over SSH, for managing one or more physical and virtual machines running :ref:`Ubuntu`. * https://github.com/juju/juju/issues/470 .. index:: osquery .. _osquery: osquery ~~~~~~~~ | Homepage: https://osquery.io/ | Src: https://github.com/facebook/osquery | Docs: https://osquery.io/docs/tables/ | Docs: https://osquery.readthedocs.io/en/stable/ osquery is a tool for reading and querying many sources of system data with SQL for :ref:`OSX` and :ref:`Linux`. * https://docs.saltstack.com/en/develop/ref/modules/all/salt.modules.osquery.html * https://github.com/westurner/dotfiles/blob/develop/scripts/osquery-all.sh .. index:: Puppet .. _puppet: Puppet ~~~~~~~~ | Wikipedia: ``__ | Homepage: https://puppetlabs.com/ | Docs: https://puppet.com/docs | Docs: https://puppet.com/docs/puppet/ | Src: git https://github.com/puppetlabs | TcpPort: 8140 Puppet is a :ref:`Configuration Management` system written in :ref:`Ruby` which runs Puppet Modules written in Puppet DSL or :ref:`Ruby` for managing one or more physical and virtual machines running various operating systems. * https://github.com/nanliu/puppet-transport .. index:: Salt .. _salt: Salt ~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.saltstack.com | Src: git https://github.com/saltstack/salt | Docs: https://docs.saltstack.com/en/latest/ | Docs: https://docs.saltstack.com/en/latest/salt-modindex.html | Docs: https://docs.saltstack.com/en/latest/ref/states/all/index.html | Docs: https://docs.saltstack.com/en/latest/ref/clients/index.html#python-api | Docs: https://docs.saltstack.com/en/latest/topics/development/hacking.html | Docs: https://docs.saltstack.com/en/latest/glossary.html | Pypi: https://pypi.python.org/pypi/salt | Twitter: https://twitter.com/SaltStackInc | IRC: ``#salt`` | TcpPort: 4505 (salt zmq) | TcpPort: 4506 (salt zmq) | TcpPort: 22 (salt-ssh) Salt is a :ref:`Configuration Management` system written in :ref:`Python` which runs Salt Formulas written in :ref:`YAML`, :ref:`Jinja2`, :ref:`Python` for managing one or more physical and virtual machines running various operating systems. .. index:: Software Build Tools .. index:: Build Automation Tools .. _software build tools: .. _build automation tools: Build Automation Tools ======================= | Wikipedia: https://en.wikipedia.org/wiki/Build_automation - https://en.wikipedia.org/wiki/List_of_build_automation_software .. index:: Autotools .. _autoconf: .. _automake: .. _autotools: GNU Autotools ~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNU_Autotools | Homepage: https://www.gnu.org/software/automake/ | Homepage: https://www.gnu.org/software/automake/ | Docs: https://www.gnu.org/software/autoconf/manual/autoconf.html | Docs: https://www.gnu.org/software/automake/manual/automake.html | Docs: https://www.gnu.org/software/automake/manual/automake.html#Standard-Configuration-Variables GNU Autotools (*GNU Build System*) are a set of tools for software build automation: autoconf, automake, libtool, and gnulib. - The traditional ``./configure --help; make; make install`` build workflow comes from the GNU Build System. - Autoconf uses a ``configure.ac`` configure *include* file in generating a ``configure`` script that checks for platform and software dependencies and caches the results in a ``config.status`` script, which generates ``config.h`` C header file that caches the results - Automake uses a ``Makefile.am`` to generate a ``Makefile.in`` makefile *include* file that generates a :ref:`GNU Make` ``Makefile`` - GNU Coding Standards define a number of standard configuration variables: ``CC``, ``CFLAGS``, ``CXX``, ``CXXFLAGS``, ``LDFLAGS``, ``CPPFLAGS`` which tools such as :ref:`GNU Make` automatically add to e.g. :ref:`GCC` (and :ref:`ld`) build program arguments .. code:: bash $ # autoconf # configure.ac -> ./configure $ ./configure --help $ ./configure --prefix=/usr/local --with-this-or-that make .. index:: Bake .. _bake: Bake ~~~~~~ | Src: https://github.com/kennethreitz/bake Bake is a free and open source software build automation tool similar in form and function to :ref:`Make`. - Bake uses ``Bakefile`` files to describe builds. - Bakefiles can contain e.g. :ref:`Bash` and :ref:`Python` scripts (instead of :ref:`Make` syntax) .. index:: BUILD .. _build: BUILD ~~~~~~~ A number of tools use (incompatible) ``BUILD`` files to describe software builds: - Google :ref:`Blaze` - :ref:`Bazel` - Twitter :ref:`Pants Build` .. index:: Google Blaze .. index:: Blaze .. _google blaze: .. _blaze: Blaze ~~~~~~~ Blaze is an internal software build automation tool developed by Google. - Blaze was the first build tool to use :ref:`BUILD` files. - :ref:`Bazel` is an open source rewrite of Blaze. .. index:: Bazel .. _bazel: Bazel ~~~~~~~ | Wikipedia: ``__ | Homepage: https://bazel.build/ | Src: https://github.com/bazelbuild/bazel | Docs: https://docs.bazel.build/ | Docs: https://docs.bazel.build/versions/1.2.0/build-ref.html Bazel is a free and open source software build automation tool developed as a rewrite of Google :ref:`Blaze`. - A ``WORKSPACE`` or ``WORKSPACE.bazel`` file indicates the root of a Bazel workspace. - Bazel uses :ref:`BUILD` (or ``BUILD.bazel``) files to describe builds. - :ref:`Buck` was released before :ref:`Bazel` was open-sourced. .. index:: BuckBuild .. index:: Buck .. _buck: Buck ~~~~~~ | Wikipedia: ``__ | Homepage: https://buck.build/ | Src: https://github.com/facebook/buck | Docs: https://buck.build/setup/getting_started.html | Docs: https://buck.build/about/overview.html - Buck uses ``BUCK`` files to describe builds. .. index:: CMake .. _cmake: CMake ~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/CMake | Homepage: https://cmake.org/ | Src: https://gitlab.kitware.com/cmake/cmake | Docs: https://cmake.org/cmake/help/latest/ | Docs: https://cmake.org/cmake/help/latest/guide/tutorial/ CMake is a free and open source software build automation tool. - CMake generates build configurations for a number of tools: Unix Makefiles, Ninja, Visual Studio .. index:: Gradle .. _gradle: Gradle ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Gradle | Homepage: https://gradle.org/ | Src: git https://github.com/gradle/gradle | Download: https://gradle.org/downloads | Docs: https://docs.gradle.org/current/release-notes | Docs: https://docs.gradle.org/current/userguide/userguide.html | Twitter: https://twitter.com/gradle Gradle is a build tool for the :ref:`Java` :ref:`JVM` which builds a directed acyclic graph (DAG). .. index:: Grunt .. _grunt: Grunt ~~~~~~ | Homepage: https://gruntjs.com/ | Src: git https://github.com/gruntjs/grunt | Docs: https://gruntjs.com/getting-started | Docs: https://gruntjs.com/plugins | Twitter: https://twitter.com/gruntjs Grunt is a build tool written in :ref:`Javascript` which builds a directed acyclic graph (DAG). .. index:: Gulp .. _gulp: Gulp ~~~~~ | Homepage: https://gulpjs.com/ | Src: https://github.com/gulpjs/gulp | Docs: https://github.com/gulpjs/gulp/blob/master/docs/ | Docs: https://github.com/gulpjs/gulp/blob/master/docs/getting-started.md | Docs: https://gulpjs.com/plugins/ | Twitter: https://twitter.com/gulpjs Gulp is a build tool written in :ref:`Javascript` which builds a directed acyclic graph (DAG). .. index:: Jake .. _jake: Jake ~~~~~ | Homepage: https://jakejs.com/ | Src: git https://github.com/jakejs/jake | NPMPkg: jake | NPM: https://www.npmjs.com/package/jake | Docs: https://jakejs.com/docs-page.html Jake is a :ref:`Javascript` build tool written in :ref:`Javascript` (for :ref:`Node.js`) similar to :ref:`Make` or :ref:`Rake`. .. index:: Make .. _make: Make ~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.gnu.org/software/make/ | Project: https://savannah.gnu.org/projects/make/ | Src: git git://git.savannah.gnu.org/make.git | Docs: https://www.gnu.org/software/make/manual/make.html GNU Make is a classic, ubiquitous software build automation tool designed for file-based source code compilation which builds a directed acyclic graph (DAG). :ref:`Bash`, :ref:`Python`, and the GNU/:ref:`Linux` kernel are all built with Make. Make build task chains are represented in a ``Makefile``. Pros * Simple, easy to read syntax * Designed to build files on disk (see: ``.PHONY``) * Nesting: ``make -C `` * Variable Syntax: ``$(VARIABLE_NAME)`` or ``${VARIABLE_NAME}`` * Bash completion: ``make `` * Python: Initially parseable with *disutils.text_file* * Logging: command names and values print to stdout (unless prefixed with ``@``) Cons * Platform Portability: make is not installed everywhere * Global Variables: parametrization with shell scripts .. code:: bash VARIABLE_NAME="value" make test make test VARIABLE_NAME="value" # ... export VARIABLE_NAME="value" make test .. index:: Pants .. index:: Pants Build .. _pants build: .. _pants: Pants Build ~~~~~~~~~~~ | Homepage: https://pantsbuild.github.io/ | Src: git https://github.com/pantsbuild/pants | Docs: https://pantsbuild.github.io/first_concepts.html | Docs: https://pantsbuild.github.io/install.html | Docs: https://pantsbuild.github.io/first_tutorial.html | Docs: https://pantsbuild.github.io/JVMProjects.html | Docs: https://pantsbuild.github.io/scala.html | Docs: https://pantsbuild.github.io/python-readme.html | Docs: https://pantsbuild.github.io/pex_design.html | Docs: https://pantsbuild.github.io/build_files.html | Docs: https://pantsbuild.github.io/build_dictionary.html | Docs: https://pantsbuild.github.io/options_reference.html | Docs: https://pantsbuild.github.io/export.html | Docs: https://pantsbuild.github.io/internals.html | Docs: https://pantsbuild.github.io/howto_contribute.html Pants Build is a build tool for :ref:`JVM` [:ref:`Java`, :ref:`Scala`, :ref:`Android`], :ref:`C++`, :ref:`Go`, :ref:`Haskell`, :ref:`Node`, and :ref:`Python` [:ref:`CPython`] software projects. * A Pants :ref:`BUILD` file defines one or more build targets: + Pants can build *Deployable Bundles*, *Runnable Binaries*, *Importable Code*, *Tests*, and *Generated Code* (e.g. Java from :ref:`Thrift` ``.thrift`` definitions). + Pants can build :ref:`PEX` files (*Python EXecutables*); which are essentially executable ZIP files with inlined dependency sets. + Pants can build :ref:`DEX` files (:ref:`Android` *Dalvik Executables*) + https://pantsbuild.github.io/build_files.html + https://pantsbuild.github.io/build_dictionary.html + https://pantsbuild.github.io/options_reference.html * A Pants ``pants.ini`` file in a top-level source directory defines options for binaries, tools, goals, tasks (sub-goals) + https://pantsbuild.github.io/setup_repo.html#configuring-with-pantsini + https://pantsbuild.github.io/options_reference.html * :ref:`Vim` plugin for Pants Build ``BUILD`` syntax: https://github.com/pantsbuild/vim-pants * :ref:`IntelliJ` plugin for Pants Build: https://github.com/pantsbuild/intellij-pants-plugin .. index:: Virtualization .. _virtualization: Virtualization =============== .. index:: Cgroups .. _cgroups: Cgroups ~~~~~~~~ | Wikipedia: ``__ | Docs: https://www.kernel.org/doc/Documentation/cgroups/ | Docs: https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/ | Docs: https://docs.fedoraproject.org/en-US/Fedora/17/html-single/Resource_Management_Guide/index.html#sec-How_Control_Groups_Are_Organized | Docs: https://wiki.archlinux.org/index.php/Cgroups Cgroups are a :ref:`Linux` mechanism for containerizing groups of processes and resources. * https://chimeracoder.github.io/docker-without-docker/#1 * ``systemd-nspawn``, ``systemd-cgroup`` * ``machinectl``, ``systemctl``, ``journalctl``, .. index:: libcontainer .. _libcontainer: libcontainer ~~~~~~~~~~~~~ | Homepage: https://www.opencontainers.org/ | Src: https://github.com/docker/libcontainer | Src: https://github.com/opencontainers/runc/tree/master/libcontainer libcontainer is a library built by :ref:`Docker` to replace :ref:`LXC`. Libcontainer provides a native :ref:`Go` implementation for creating containers with namespaces, :ref:`cgroups`, capabilities, and filesystem access controls. -- https://github.com/opencontainers/runc/tree/master/libcontainer - libcontainer is now developed as part of :ref:`OCI` :ref:`runC`. .. index:: OCI .. index:: Open Container Initiative .. _open container initiative: .. _oci: Open Container Initiative ~~~~~~~~~~~~~~~~~~~~~~~~~~ | Homepage: https://www.opencontainers.org/ | Docs: https://www.opencontainers.org/about | Src: https://github.com/opencontainers | Twitter: https://twitter.com/oci_org The Open Container Initiative (*OCI*) is a Linux Foundation collaborative project dedicated to developing a working, portable software container specification. .. index:: runC .. _runc: runC ~~~~~~ | Homepage: https://www.opencontainers.org/ | Src: https://github.com/opencontainers/runc runC is a container abstraction runc is a CLI tool for spawning and running containers according to the :ref:`OCI` specification. - runC builds upon the :ref:`libcontainer` abstraction and the :ref:`OCI` container specification. - runC works on :ref:`Linux`, :ref:`OSX`, and :ref:`Windows`. - runC containers do not require a daemon process. - runC containers can run as e.g. a systemd service unit. .. index:: Docker .. _docker: Docker ~~~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.docker.com/ | Src: https://github.com/docker/docker | Docs: https://docs.docker.com/ | Awesome: https://github.com/veggiemonk/awesome-docker Docker is an OS virtualization project written in :ref:`Go` which utilizes :ref:`Linux` containers -- first :ref:`LXC` now :ref:`libcontainer` / :ref:`runC` -- to partition process workloads across one or more host systems. .. glossary:: Dockerfile A ``Dockerfile`` contains the instructions needed to create a docker image. Docker container A Docker container is an instance of a :term:`Docker Image` with configuration. Docker API The Docker API is an interface of management commands for provisioning and managing containers. :term:`Docker Machine`, :term:`Docker Swarm`, and :term:`Docker Universal Control Plane` all implement the Docker API; so the ``docker`` client works equally well with each implementation. Docker Machine Docker Machine is the container management application which implements the :term:`Docker API`. Docker Swarm Docker Swarm is a cluster management system for Docker containers hosted on one or more :term:`Docker Machines ` Docker Universal Control Plane Docker Universal Control Plane is an enterprise-grade cluster management solution with a web dashboard and external authentication which implements the :term:`Docker API`. Docker Compose Docker Compose is a Python application for defining and managing services (:term:`Docker containers `) and networks with a ``docker-compose.yml`` :ref:`YAML` configuration file. Docker Image A Docker Image is an archived container filesystem with configuration which is usually defined by a :term:`Dockerfile`. Docker Hub Docker Hub is a cloud-based registry service for :term:`Docker Images `. Docker Cloud Docker Cloud is the hosting service offered by Docker. * Docker images build from a :term:`Dockerfile` * A ``Dockerfile`` can subclass another Dockerfile (to add, remove, or change configuration) * ``Dockerfile`` support a limited number of commands * Docker is not intended to be a complete :ref:`configuration management system ` * Ideally, a Docker image requires minimal configuration once built * Docker images can be hosted by https://hub.docker.com/ * ``docker run -it ubuntu/16.04`` downloads the image from https://hub.docker.com/_/ubuntu/, creates a new instance (``docker ps``), and spawns a root :ref:`Shell ` with a UUID name (by default). * There are a number of ways to "Schedule" [redundant] persistent containers that launch on boot with :ref:`Docker` - Docker Swarm is the Docker-native way to run a cluster of containers. To a client app, Docker Swarm looks just like Docker Machine because it implements the Docker API. - :ref:`Kubernetes` is one project which uses Docker to schedule redundant, optionally geodistributed, :ref:`LXC` containers (in "Pods"). :ref:`Salt` can install and manage docker, docker images and containers: * https://github.com/saltstack-formulas/docker-formula * https://docs.saltstack.com/en/latest/ref/states/all/salt.states.dockerio.html * https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.dockerio.html .. index:: CNCF .. index:: Cloud Native Computing Foundation .. _cloud native computing foundation: .. _cncf: Cloud Native Computing Foundation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Linux_Foundation#Cloud_Native_Computing_Foundation The Cloud Native Computing Foundation (*CNCF*) is a foundation for cloud and container industry collaboration. - :ref:`Kubernetes` is now a CNCF project. .. index:: Kubernetes .. _kubernetes: Kubernetes ~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Kubernetes | Homepage: https://kubernetes.io/ | Src: https://github.com/GoogleCloudPlatform/kubernetes | Docs: https://kubernetes.io/gettingstarted/ | Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/docker.md | Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md | Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos.md | Docs: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md | Q&A: https://stackoverflow.com/questions/tagged/kubernetes | Twitter: https://twitter.com/kubernetesio | Awesome: https://github.com/ramitsurana/awesome-kubernetes Kubernetes (*k8s*) is a highly-available distributed cluster scheduler which works with groups of :ref:`Docker` containers called Pods. - Google donated Kubernetes to the :ref:`CNCF`. .. index:: k3s .. _k3s: k3s ~~~~ | Homepage: https://k3s.io/ | Src: https://github.com/rancher/k3s | Docs: https://rancher.com/docs/k3s/latest/en/ k3s is a lightweight :ref:`Kubernetes` distribution which runs on x86-64, ARM6, ARM7; only requires 512Mb of RAM; and is distributed as a single :ref:`Go` binary. - You've heard of k8s? This is k8s - 5. .. index:: Kubernetes-Mesos .. _kubernetes-mesos: Kubernetes-Mesos ~~~~~~~~~~~~~~~~~ | Src: https://github.com/mesosphere/kubernetes-mesos kubernetes-mesos integrates :ref:`Kubernetes` :ref:`Docker` Pod scheduling with :ref:`Mesos`. .. epigraph:: Kubernetes and Mesos are a match made in heaven. Kubernetes enables the Pod, an abstraction that represents a group of co-located containers, along with Labels for service discovery, load-balancing, and replication control. Mesos provides the fine-grained resource allocations for pods across nodes in a cluster, and facilitates resource sharing among Kubernetes and other frameworks running on the same cluster. .. index:: KVM .. _KVM: KVM ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine | Homepage: https://www.linux-kvm.org/ | Docs: https://www.linux-kvm.org/page/Documents KVM is a full virtualization platform with support for Intel VT and AMD-V; which supports running various guest operating systems, each with their own kernel, on a given host machine. .. index:: Libcloud .. _libcloud: Libcloud ~~~~~~~~~~~~~~~~~~ | Homepage: https://libcloud.apache.org/ | Docs: https://libcloud.readthedocs.io/en/latest/ | Docs: https://libcloud.readthedocs.io/en/latest/supported_providers.html | Src: git git://git.apache.org/libcloud.git | Src: git https://github.com/apache/libcloud Apache libcloud is a :ref:`Python` library which abstracts and unifies a large number of Cloud APIs for Compute Resources, Object Storage, Load Balancing, and DNS. :ref:`Salt` :term:`salt cloud` depends upon libcloud. .. index:: Libvirt .. _libvirt: Libvirt ~~~~~~~~~~~~~~~~~ | Wikipedia: https://libvirt.org/ | Homepage: https://libvirt.org/ | Docs: https://libvirt.org/docs.html | Docs: https://libvirt.org/cgroups.html | Docs: https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.virt.html | Src: git git://libvirt.org/libvirt-appdev-guide.git Libvirt is a system for platform virtualization with various :ref:`Linux` hypervisors. * Xen * QEMU, :ref:`KVM` * OpenVZ, :ref:`LXC` * :ref:`VirtualBox` .. index:: LXC .. _LXC: LXC ~~~~ | Wikipedia: https://en.wikipedia.org/wiki/LXC | Homepage: https://linuxcontainers.org/ | Docs: https://linuxcontainers.org/lxc/documentation/ | Src: https://github.com/lxc/lxc LXC ("Linux Containers"), written in :ref:`C`, builds upon :ref:`Linux` :ref:`Cgroups` to provide containerized OS chroots (all running under :ref:`the host kernel `). LXC is included in recent :ref:`Linux` kernels. .. index:: LXD .. _LXD: LXD ~~~~ | Homepage: https://linuxcontainers.org/lxd/ | Docs: https://linuxcontainers.org/lxd/ | Src: https://github.com/lxc/lxd LXD, written in :ref:`Go`, builds upon :ref:`LXC` to provide a system-wide daemon and an :ref:`OpenStack` Nova hypervisor plugin. .. index:: Mesos .. _mesos: Mesos ~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Apache_Mesos | Homepage: https://mesos.apache.org/ | Src: git https://github.com/apache/mesos | Q&A: https://stackoverflow.com/tags/mesos | Twitter: https://twitter.com/ApacheMesos Apache Mesos is a highly-available distributed datacenter operating system, for which there are many different task/process/service schedulers. .. epigraph:: Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. .. index:: Mesosphere .. _mesosphere: Mesosphere ~~~~~~~~~~~~~ | Homepage: https://mesosphere.com/ | Src: https://github.com/mesosphere | Q&A: https://stackoverflow.com/tags/mesosphere | Twitter: https://twitter.com/mesosphere * Apache :ref:`Mesos` is a core Mesosphere service .. index:: OpenStack .. _openstack: OpenStack ~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/OpenStack | Homepage: https://www.openstack.org/ | Src: https://git.openstack.org | Src: https://github.com/openstack | Q&A: https://stackoverflow.com/questions/tagged/openstack | Docs: https://docs.openstack.org/ | Docs: https://wiki.openstack.org/ | Docs: https://wiki.openstack.org/wiki/Get_OpenStack | Twitter: https://twitter.com/openstack OpenStack is a platform of infrastructure services for running a cloud datacenter (a *private* or a *public* cloud). * OpenStack can be installed on one machine with enough RAM, or many thousands of machines. * OpenStack **Keystone** -- cluster/grid/cloud-level token and user-service based authentication (authN) and authorization (authZ) as a service. * Src: https://github.com/openstack/keystone * Wiki: https://wiki.openstack.org/wiki/Keystone * OpenStack **Nova** implements a Hypervisor API which abstracts various :ref:`Virtualization` providers (e.g. :ref:`KVM`, :ref:`Docker`, :ref:`LXC`, :ref:`LXD`). * Src: https://github.com/openstack/nova * Wiki: https://wiki.openstack.org/wiki/Nova * Docs: https://wiki.openstack.org/wiki/HypervisorSupportMatrix * Docs: https://docs.openstack.org/nova/latest/user/support-matrix.html * OpenStack **Swift** -- redundant HTTP-based Object Storage as a service. * Src: https://github.com/openstack/swift * Wiki: https://wiki.openstack.org/wiki/Swift * Docs: https://docs.openstack.org/swift/latest/ * Docs: https://docs.openstack.org/swift/latest/overview_auth.html * Docs: https://docs.openstack.org/swift/latest/overview_object_versioning.html * OpenStack **Neutron** (*Quantum*)-- software defined networking (SDN), VLAN, switch configuration, virtual and physical enterprise networking as a service. * Src: https://github.com/openstack/neutron * Wiki: https://wiki.openstack.org/wiki/Neutron * OpenStack **Designate** -- DNS as a service (Bind9, PowerDNS) integrated with OpenStack Keystone, Neutron, and Nova. * Src: https://github.com/openstack/designate * Wiki: https://wiki.openstack.org/wiki/Designate * OpenStack **Poppy** -- CDN as a service CDN vendor API * Src: https://github.com/stackforge/poppy * Wiki: https://wiki.openstack.org/wiki/Poppy * OpenStack **Horizon** -- web-based OpenStack Dashboard which is written in Django. * Src: https://github.com/openstack/horizon * https://wiki.openstack.org/wiki/Horizon OpenStack makes it possible for end-users to create a new virtual machine from the available pool of resources. ``rdfs:seeAlso``: :ref:`openstack-devstack`, :ref:`Libcloud` .. index:: OpenStack DevStack .. _openstack-devstack: OpenStack DevStack ~~~~~~~~~~~~~~~~~~ | Src: https://github.com/openstack-dev/devstack | Docs: https://docs.openstack.org/devstack/latest/ | Docs: https://docs.openstack.org/devstack/latest/overview.html | Issues: https://launchpad.net/devstack OpenStack DevStack is a default development configuration for :ref:`OpenStack`. * https://github.com/openstack-dev/devstack-vagrant There are many alternatives to and implementations of OpenStack DevStack: * https://github.com/saltstack-formulas/openstack-standalone-formula * https://github.com/CSSCorp/openstack-automation * https://github.com/openstack-ansible/openstack-ansible * https://forge.puppetlabs.com/puppetlabs/openstack * https://jujucharms.com/q/openstack * https://anvil.readthedocs.io/en/latest/topics/summary.html .. index:: Packer .. _packer: Packer ~~~~~~~~~~~~~~~~~ | Homepage: https://www.packer.io/ | Src: https://github.com/hashicorp/packer | Docs: https://www.packer.io/docs | Docs: https://www.packer.io/docs/basics/terminology.html Packer generates machine images for multiple platforms, clouds, and hypervisors from a parameterizable template. .. glossary:: Packer Artifact Build products: machine image and manifest Packer Template JSON build definitions with optional variables and templating Packer Build Task defined by a JSON file containing build steps which produce a machine image Packer Builder Packer components which produce machine images for one of many platforms: - :ref:`VirtualBox` - :ref:`Docker` - :ref:`OpenStack` - GCE - EC2 - VMware - QEMU (:ref:`KVM`, Xen) - https://www.packer.io/docs/templates/builders.html Packer Provisioner Packer components for provisioning machine images at build time - Shell scripts - File uploads - ansible - chef - solo - puppet - salt Packer Post-Processor Packer components for compressing and uploading built machine images .. index:: Vagrant .. _vagrant: Vagrant ~~~~~~~~~~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://www.vagrantup.com/ | Docs: https://www.vagrantup.com/docs/ | Src: https://github.com/hashicorp/vagrant Vagrant is a tool written in :ref:`Ruby` for creating and managing virtual machine instances with CPU, RAM, Storage, and Networking. * Vagrant: * Works with a number of `Cloud` and :ref:`Virtualization` providers: * :ref:`VirtualBox` * AWS EC2 * GCE * :ref:`OpenStack` * provides helpful commandline porcelain on top of :ref:`VirtualBox` ``VboxManage`` * installs and lifecycles *Vagrant Boxes* :: vagrant help vagrant status vagrant init ubuntu/trusty64 vagrant up vagrant ssh $EDITOR Vagrantfile vagrant provision vagrant halt vagrant destroy .. glossary:: vagrantfile vagrant script defining a team of one or more virtual machines and networks. create a vagrantfile:: vagrant init [basebox] cat vagrantfile start virtual machines and networks defined in the vagrantfile:: vagrant status vagrant up Vagrant Box Vagrant base machine virtual machine image. There are many baseboxes for various operating systems. Essentially a virtual disk plus CPU, RAM, Storage, and Networking metadata. Locally-stored and cached vagrant boxes can be listed with:: vagrant help box vagrant box list A running vagrant environment can be packaged into a new box with:: vagrant package :ref:`Packer` generates :ref:`VirtualBox` Vagrant Boxes with a Post-Processor. Vagrant Cloud Vagrant-hosted public Vagrant Box storage. Install a box from Vagrant cloud:: vagrant init ubuntu/trusty64 vagrant up vagrant ssh Vagrant Provider A driver for running Vagrant Boxes with a hypervisor or in a cloud. The Vagrant :ref:`VirtualBox` Provider is well-supported. With Plugins: https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins See also: :ref:`libcloud`. Vagrant Provisioner Set of hooks to install and run shell scripts and configuration managment tools over ``vagrant ssh``. Vagrant up runs ``vagrant provision`` on first invocation of ``vagrant up``. :: vagrant provision .. note:: Vagrant configures a default NFS share mounted at ``/vagrant``. .. note:: Vagrant adds a default NAT Adapter as eth0; presumably for DNS, the default route, and to ensure ``vagrant ssh`` connectivity. .. index:: VirtualBox .. _virtualbox: VirtualBox ~~~~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/VirtualBox | Homepage: https://www.virtualbox.org/ | Docs: https://www.virtualbox.org/wiki/Documentation | Src: svn svn://www.virtualbox.org/svn/vbox/trunk Oracle VirtualBox is a platform virtualization package for running one or more guest VMs (virtual machines) within a host system. VirtualBox: * runs on many platforms: :ref:`Linux`, OSX, Windows * has support for full platform NX/AMD-v virtualization * requires matching kernel modules :ref:`Vagrant` scripts VirtualBox. .. index:: Shells .. _shells: Shells ======== .. index:: Bash .. _bash: Bash ~~~~~~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.gnu.org/software/bash/ | Src: git git://git.savannah.gnu.org/bash.git | Docs: https://www.gnu.org/software/bash/manual/ | LearnXinYMinutes: https://learnxinyminutes.com/docs/bash/ | Awesome: https://github.com/awesome-lists/awesome-bash GNU Bash, the Bourne-again shell, is an open source command-line program written in :ref:`C` for running commands in a text-based terminal. A few commands to try when learning to shell with Bash: .. code-block:: bash echo $SHELL; echo "$SHELL"; echo "${SHELL}" type bash bash --help help help help type apropos bash info bash man bash man man info info # [down arrow] and then [enter] to select, or 'n' for next * Bash works with unix command outputs and return codes: a program returns nonzero when there is an error: .. code:: bash true; echo $? # 0 false; echo $? # 1 echo "Hello" && echo " World!" # Hello World! false || echo "World!" # World! * Functions: Bash supports functions with arguments that can print to standard out and/or return an integer return code: .. code:: bash function add_a { echo "$1 + $2 = $(( $1 + $2 ))" } add_b () { echo "$1 + $2 = $(( $1 + $2 ))" } add_xy () { echo "$x + $y = $(( $x + $y ))" } add_a 3 5 # "3 + 5 = 8" add_b 3 5 # "3 + 5 = 8" x=3 y=5 add_xy # "3 + 5 = 8" x=3; y=5; add_xy # "3 + 5 = 8" output=$(add_a 3 5) echo "${output}" help test help [ help [[ help return test "$(add_a 3 5)" == "3 + 5 = 8" && echo 'OK' test_add_a () { if [[ "$(add_a 3 5)" == "3 + 5 = 8" ]]; then echo 'OK' return 0 else echo 'Test failed' return 1 fi } test_add_a help trap help exit * Portability: sh (sh, bash, dash, zsh) shell scripts are mostly compatible; though bash supports some features that other shells do not. * Logging: You can configure bash to print commands and arguments as bash executes scripts: .. code:: bash set -x # print commands and arguments set -v # print source Bash reads various configuration files at startup time: .. code:: bash /etc/profile /etc/bash.bashrc /etc/profile.d/*.sh ${HOME}/.profile /etc/skel/.profile # PATH=+$HOME/bin # umask ${HOME}/.bash_profile # empty. preempts .profile ${HOME}/.bashrc Bash and various :ref:`Operating Systems`: - Linux: Bash is almost always installed as the default shell on Linux boxes. - Mac: - MacOS includes Bash 3.2. - You can ``brew install bash`` to get a more recent version. (:ref:`homebrew`) - Windows: - :ref:`Windows Subsystem for Linux` (WSL) installs Linux distributions which include bash. - You can also install bash on Windows by installing git with ``choco install git -y`` (:ref:`Chocolatey`) - You can also install bash on Windows by installing MSYS2 (Mingw) or Cygwin with ``choco install msys2`` or ``choco install cygwin`` While Bash is ubiquitous, shell scripts are loose with quoting; which makes shell scripts flexible but **dangerous** and thus often avoided in favor of other languages: .. code:: bash ## Shell script quoting example 1: # This prints a newline echo $(echo "-e a\nb") # This prints "-e a\nb" echo "$(echo "-e a\nb")" This isn't an issue with e.g. :ref:`Python` (a popular language that's also useful for system administration). .. code:: python import subprocess print(subprocess.check_output(['echo', "-e a\nb"]) print(subprocess.check_output('echo "-e a\nb"', shell=True)) # Though, note that Python subprocess shell=True is a security risk: # - avoid shell=True # - pass the command as a list of already-tokenized arguments # - use something like sarge (or ansible) instead of shell=True :ref:`IPython` is one of many alternatives to Bash. .. index:: Readline .. _readline: Readline ~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNU_Readline | Homepage: https://tiswww.case.edu/php/chet/readline/rltop.html | Docs: https://tiswww.case.edu/php/chet/readline/readline.html | Docs: https://tiswww.case.edu/php/chet/readline/history.html | Docs: https://tiswww.case.edu/php/chet/readline/rluserman.html | Src: ftp ftp://ftp.gnu.org/gnu/readline/readline-8.0.tar.gz | Pypi: https://pypi.python.org/pypi/gnureadline .. index:: IPython .. index:: ipython .. _ipython: IPython ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/IPython | Homepage: https://ipython.org/ | Src: git https://github.com/ipython/ipython | DockerHub: https://hub.docker.com/repos/ipython/ | Docs: https://ipython.readthedocs.io/en/stable/ | Docs: https://ipython.readthedocs.io/en/stable/interactive/ | Docs: https://ipython.readthedocs.io/en/stable/parallel/ | Docs: https://github.com/ipython/ipython/wiki/Extensions-Index | Docs: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels | Docs: https://github.com/ipython/ipython/wiki/Install:-Docker IPython is an interactive REPL and distributed computation framework written in :ref:`Python`. .. code:: bash ## Formatting expression output with the Python interpreter 1 + 1 x = 1+1 print("1 + 1 = 2") print('1 + 1 = %d' % (x)) print('1 + 1 = {0}'.format(x)) # Python 2.7+ print('1 + 1 = {x}'.format(x=x)) # Python 2.7+ print(f'1 + 1 = {x}') # Python 3.6+ print(f'{1 + 1 = }') # Python 3.8+ ## IPython !ipython --help # run `$SHELL -c 'ipython --help'` !python -m IPython --help # run `ipython --help` ? # print IPython help within IPython %lsmagic % # list magic commands and aliases %paste? # help for the %paste magic command %logstart? # help for the %logstart magic command %logstart -o logoutput.log.py # log input and output to a file import json json? # print(json.__doc__) json?? # print(inspect.getsource(json)) ## IPython shell !cat ./README.rst; echo $PWD # run shell commands lines = !ls -al # capture shell command output print(lines[0:]) %run -i -t example.py # run a script with timing info, # in the local namespace %run -d example.py # run a script with pdb %pdb on # automatically run pdb on Exception - If a kernel is not specified, IPython uses the ``ipykernel`` Jupyter kernel. - To use other kernels with IPython, you must install `jupyter_console` and a kernel: .. code:: bash pip install jupyter_console # conda install -y jupyter_console ipython console --kernel python # ipykernel jupyter console --kernel python # ipykernel # | | "exit()" pip install bash_kernel # conda install -y bash_kernel jupyter console --kernel bash # "exit" conda install -y -c conda-forge xeus-cling jupyter console --kernel xcpp11 # ( on Windows) conda install -y nodejs; npm install -g ijavascript; ijsinstall jupyter console --kernel javascript # conda install -y nodejs; npm install -g jp-babel; jp-babel-install jupyter console --kernel babel # conda install -y nodejs; npm install -g itypescript; its --install=local jupyter console --kernel typescript # jupyter kernelspec list - There are very many Jupyter kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels - :ref:`Jupyter Notebook` and :ref:`Jupyter Lab` are built atop IPython: a Jupyter notebook file is a :ref:`JSON` file with an `.ipynb` extension which contains inputs and text and binary outputs. .. index:: PowerShell .. index:: Windows PowerShell .. _powershell: PowerShell ~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Windows_PowerShell | Homepage: https://microsoft.com/powershell Windows PowerShell is a shell for :ref:`Windows`. .. index:: ZSH .. _zsh: ZSH ~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Z_shell | Homepage: https://www.zsh.org/ | Docs: http://zsh.sourceforge.net/Guide/zshguide.html | Docs: http://zsh.sourceforge.net/Doc/ | Src: git git://git.code.sf.net/p/zsh/code * https://github.com/robbyrussell/oh-my-zsh .. index:: Shell Utilities .. shell-utilities: Shell Utilities ================= .. index:: Awk .. _awk: Awk ~~~~~ | Wikipedia: ``__ | Docs: https://en.wikipedia.org/wiki/AWK#Versions_and_implementations | GNU Awk | Project: https://savannah.gnu.org/projects/gawk/ | Homepage: https://www.gnu.org/software/gawk/ | Docs: https://www.gnu.org/software/gawk/manual/ | Docs: https://www.gnu.org/software/gawk/manual/gawk.html | Src: git git://git.savannah.gnu.org/gawk.git AWK is a pattern programming language for matching and transforming text. .. index:: Grep .. _grep: Grep ~~~~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.gnu.org/software/grep/ | Project: https://savannah.gnu.org/projects/grep/ | Docs: https://www.gnu.org/software/grep/manual/ | Docs: https://www.gnu.org/software/grep/manual/grep.html | Src: git git://git.savannah.gnu.org/grep.git Grep is a commandline utility for pattern-based text matching. .. index:: Htop .. _htop: Htop ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Htop | Homepage: https://hisham.hm/htop/ | Src: https://github.com/hishamhm/htop Htop is a commandline task manager; like ``top`` extended. .. index:: Pyline .. _pyline: Pyline ~~~~~~~~ | Homepage: https://github.com/westurner/pyline | Docs: https://pyline.readthedocs.io/en/latest/ | Src: git https://github.com/westurner/pyline | Pypi: https://pypi.python.org/pypi/pyline Pyline is an `open source` :ref:`POSIX` command-line utility for streaming line-based processing in :ref:`Python` with regex and output transform features similar to :ref:`grep`, :ref:`sed`, and :ref:`awk`. * Pyline can generate quoted CSV, :ref:`JSON `, HTML, etc. .. index:: Pyrpo .. _pyrpo: Pyrpo ~~~~~~ | Homepage: https://github.com/westurner/pyrpo | Src: git https://github.com/westurner/pyrpo | Pypi: https://pypi.python.org/pypi/pyrpo Pyrpo is an `open source` :ref:`POSIX` command-line utility for locating and generating reports from :ref:`Git`, :ref:`Mercurial`, :ref:`Bazaar`, and :ref:`Subversion` repositories. .. index:: Sed .. _sed: Sed ~~~~~~~~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.gnu.org/software/sed/ | Project: https://savannah.gnu.org/projects/sed | Docs: https://www.gnu.org/software/sed/manual/ | Docs: https://www.gnu.org/software/sed/manual/sed.html | Docs: http://sed.sourceforge.net/ | Src: git git.savannah.gnu.org/sed.git GNU Sed is an `open source` :ref:`POSIX` command-line utility for transforming text. .. note:: BSD Sed Use ```` for explicit tabs (as ``\t`` does not work) Use ``\\\n`` or ``'$'\n`` for newlines (as ``\n`` does not work) ``sed -E`` should be consistent extended regular expressions between GNU Sed (e.g. Linux) and BSD Sed (FreeBSD, OSX). OR: ``brew install gnu-sed`` See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux See: https://superuser.com/questions/307165/newlines-in-sed-on-mac-os-x .. index:: Web Shells .. _web shells: Web Shells ============= .. index:: IPython Notebook .. index:: IPython notebook .. _ipython notebook: IPython Notebook ~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/IPython#Notebook | Src: https://github.com/ipython/ipython/tree/3.x/IPython/html | Docs: http://ipython.readthedocs.io/en/stable/notebook/ | Docs: http://ipython.readthedocs.io/en/stable/notebook/notebook.html | Docs: http://ipython.readthedocs.io/en/stable/notebook/nbformat.html | Docs: http://ipython.readthedocs.io/en/stable/notebook/nbconvert.html | Docs: http://ipython.readthedocs.io/en/stable/notebook/public_server.html | Docs: http://ipython.readthedocs.io/en/stable/notebook/security.html | Docs: https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks :ref:`IPython` Notebook (now :ref:`Jupyter Notebook`) is an open source web-based shell written in :ref:`Python` and :ref:`Javascript` for interactive and literate computing with IPython notebooks composed of raw, markdown, or code **input** and plaintext- or rich- **output** cells. * An IPython notebook (``.ipynb``) is a :ref:`JSON-` document containing input and output for a linear sequence of cells; which can be exported to many output formats (e.g. :ref:`HTML-`, RST, LaTeX, PDF); and edited through the web with IPython Notebook. * IPython Notebook is a webapp written on :ref:`tornado`, an asynchronous web application framework for Python. * seeAlso: westurner/brw (2007-)) * IPython Notebook supports :ref:`Markdown` syntax for comment cells. * :ref:`MyST Markdown` is a :ref:`commonmark` :ref:`Markdown` syntax (which can also be used to express entire Jupyter notebooks) * IPython Notebook supports more than 40 different IPython kernels for other languages: https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages * IPython Notebook development has now moved to :ref:`Jupyter` Notebook; which supports IPython kernels (and defaults to the :ref:`IPython` :ref:`CPython` :ref:`2 ` or :ref:`3 ` kernel). To start IPython Notebook (assuming the ``_SRC`` variable as defined in :ref:`Venv`): .. code:: bash pip install ipython[notebook] # pip install -e git+https://github.com/ipython/ipython@rel-3.2.1#egg=ipython # https://github.com/ipython/ipython/releases mkdir $_SRC/notebooks; cd $_SRC/notebooks ipython notebook ipython notebook --notebook-dir="${_SRC}/notebooks" # With HTTPS (TLS/SSL) ipython notebook \ --ip=127.0.0.1 \ --certfile=mycert.pem \ --keyfile=privkey.pem \ --port=8888 \ --browser=web # (optional) westurner/dotfiles/scripts/web # List supported options ipython notebook --help .. warning:: IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine. Reproducible :ref:`SciPy Stack ` IPython Notebook / :ref:`Jupyter Notebook` servers implement best practices like process isolation and privilege separation with e.g. :ref:`Docker` and/or :ref:`Jupyter` Hub. .. note:: IPython Notebook is now :ref:`Jupyter Notebook`. Jupyter Notebook runs Python notebooks with :ref:`ipykernel`, the :ref:`IPython` :ref:`Python` kernel from :ref:`IPython Notebook` .. index:: Jupyter kernels .. _jupyter kernels: Jupyter kernels ++++++++++++++++ | Src: https://github.com/ipython/ipykernel | Docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html .. index:: ipykernel .. _ipykernel: ipykernel ++++++++++ | Src: https://ipython.readthedocs.io/en/stable/install/kernel_install.html .. index:: ipython_nose .. _ipython_nose: ipython_nose ++++++++++++++ | Src: git https://github.com/taavi/ipython_nose ipython_nose is an extension for :ref:`IPython Notebook` (and likely :ref:`Jupyter Notebook`) for discovering and running test functions starting with ``test_`` (and unittest.TestCase test classes with names containing ``Test``) with `Nose `__. * ipython_nose is not (yet?) uploaded to PyPI * to install ipython_nose from GitHub (with :ref:`Pip` and :ref:`Git`): .. code:: bash pip install -e git+https://github.com/taavi/ipython_nose#egg=ipython_nose See also: * TDD: Test Driven Development https://wrdrd.github.io/docs/consulting/software-development#test-driven-development * **Refactoring** and **Reproducibility** * https://wrdrd.github.io/docs/consulting/software-development#refactoring * https://wrdrd.github.io/docs/consulting/data-science#reproducibility .. index:: nosebook .. _nosebook: nosebook ~~~~~~~~~~ | Src: https://github.com/bollwyvl/nosebook nosebook is a tool for finding and running tests in :ref:`nbformat` :ref:`IPython Notebooks ` and :ref:`Jupyter Notebooks ` with ``nose``. See also: * https://westurner.github.io/wiki/awesome-python-testing#nose .. index:: Jupyter .. index:: Project Jupyter .. _jupyter: Jupyter ~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/IPython#Project_Jupyter | Homepage: http://jupyter.org/ | Src: https://github.com/jupyter/ | DockerHub: https://hub.docker.com/repos/jupyter/ | Docs: https://jupyter.readthedocs.io/en/latest/ | Awesome: https://github.com/markusschanta/awesome-jupyter | **Awesome:** https://github.com/quobit/awesome-python-in-education#jupyter Project Jupyter expands upon components like :ref:`IPython` and :ref:`IPython Notebook` to provide a multi-user web-based shell for many languages (:ref:`Python`, :ref:`Ruby`, :ref:`Java`, :ref:`Haskell`, Julia, R). .. table:: IPython Jupyter comparison (adapted from http://jupyter.org) :class: table-striped table-responsive +-------------------------------+-------------------------------------------------------------+ | :ref:`IPython` | :ref:`Jupyter` | +-------------------------------+-------------------------------------------------------------+ | - Interactive Python shell | - Rich REPL Protocol | | - Python kernel for Jupyter | - :ref:`Jupyter Notebook` (format, environment, conversion) | | - Interactive Parallel Python | - JupyterHub (multi-user notebook server) | | | - JupyterHub authenticators (MediaWiki OAuth, GitHub OAuth) | | | - JupyterHub spawners (Docker, Sudo, Remote, Docker Swarm) | +-------------------------------+-------------------------------------------------------------+ .. index:: Jupyter Notebook .. _jupyter notebook: Jupyter Notebook +++++++++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Jupyter#Notebook | Src: https://github.com/jupyter/notebook | CondaPkg: ``notebook`` | FileExt: ``.ipynb`` | Docs: https://jupyter-notebook.readthedocs.io/en/stable/ :ref:`Jupyter` Notebook is an open source shell webapp written in :ref:`Python` and :ref:`Javascript` for interactive and literate computing with Jupyter notebooks composed of raw, markdown, or code **input** and plaintext- or rich- **output** cells. * ``.ipynb`` files are Jupyter Notebooks saved as :ref:`JSON-` documents . * An Jupyter notebook is a document containing {meta, input, and output} records for a linear sequence of cells; which can be exported to many output formats (e.g. :ref:`HTML-`, RST, LaTeX, PDF, Python, :ref:`MyST Markdown`); and edited through the web with Jupyter Notebook. * Jupyter Notebook is a webapp written on :ref:`tornado`, an asynchronous web application framework for Python. * seeAlso: westurner/brw (2007-)) * Jupyter Notebook supports :ref:`Markdown` syntax for comment cells. * :ref:`MyST Markdown` is a :ref:`commonmark` :ref:`Markdown` syntax (which can also be used to express entire Jupyter notebooks) * Jupyter Notebook supports more than 40 different Jupyter kernels for other languages: https://github.com/ipython/ipython/wiki/Jupyter-kernels-for-other-languages To start IPython Notebook (assuming the ``_SRC`` variable as defined in :ref:`Venv`): .. code:: bash pip install ipython[notebook] # pip install -e git+https://github.com/ipython/ipython@rel-3.2.1#egg=ipython # https://github.com/ipython/ipython/releases mkdir $_SRC/notebooks; cd $_SRC/notebooks ipython notebook ipython notebook --notebook-dir="${_SRC}/notebooks" # With HTTPS (TLS/SSL) ipython notebook \ --ip=127.0.0.1 \ --certfile=mycert.pem \ --keyfile=privkey.pem \ --port=8888 \ --browser=web # (optional) westurner/dotfiles/scripts/web # List supported options ipython notebook --help .. warning:: IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine. Reproducible :ref:`SciPy Stack ` IPython Notebook / :ref:`Jupyter Notebook` servers implement best practices like process isolation and privilege separation with e.g. :ref:`Docker` and/or :ref:`Jupyter` Hub. .. note:: :ref:`JupyterLab` (a mostly-rewrite) adds e.g. tabs and undo (and a new extension API) to :ref:`Jupyter Notebook`. .. index:: JupyterLab .. _jupyterlab: JupyterLab +++++++++++++ | Wikipedia: https://en.wikipedia.org/wiki/Project_Jupyter#JupyterLab | Src: https://github.com/jupyterlab/jupyterlab | CondaPkg: ``jupyterlab`` | Docs: https://jupyterlab.readthedocs.io/en/stable/ | Docs: https://jupyterlab.readthedocs.io/en/stable/user/extensions.html | Docs: https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html | Awesome: https://github.com/mauhai/awesome-jupyterlab JupyterLab is an open-source web-based tabbed :ref:`IDE ` written in :ref:`Python`, :ref:`Javascript`, and :ref:`TypeScript` for working with :ref:` Jupyter Notebook`, terminals, text editing, undo, extensions. - You can edit Jupyter notebooks with JupyterLab. - Installing JupyterLab also installs :ref:`Jupyter Notebook`. (which doesn't support tabs or the new extension API) - A few UI differences between JupyterLab and Jupyter Notebook: - JupyterLab has tabbed editing: you can open files, notebooks, and terminals in tabs - JupyterLab has a sidebar with a file selector pane - Installing JupyterLab does not install any :ref:`scipystack` or other packages. .. index:: JupyterLab (install) Install JupyterLab +++++++++++++++++++++ Install JupyterLab With :ref:`pip`: .. code:: bash python -m pip install jupyterlab Install JupyterLab with :ref:`conda`: .. code:: bash conda install -c conda-forge -y jupyterlab .. index:: Hosting JupyterLab .. index:: JupyterLab (hosting) .. _hosting jupyterlab: Hosting JupyterLab +++++++++++++++++++ You can host JupyterLab yourself: - :ref:`JupyterHub` - "Zero to JupyterHub with :ref:`Kubernetes`" https://zero-to-jupyterhub.readthedocs.io/en/latest/ - JupyterHub **Spawners** create new instances of JupyterLab (within containers, VMs, a shell) - https://jupyterhub.readthedocs.io/en/stable/reference/spawners.html - https://github.com/jupyterhub/jupyterhub/wiki/Spawners - JupyterHub **Authenticators** check names and credentials from a file, PAM, Single Sign On APIs - https://jupyterhub.readthedocs.io/en/stable/reference/authenticators.html - https://github.com/jupyterhub/jupyterhub/wiki/Authenticators - :ref:`BinderHub` runs containers with :ref:`repo2docker` - https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/ - :ref:`mybinder.org` is powered by :ref:`BinderHub` .. index:: Hosted JupyterLab .. index:: JupyterLab (hosted) .. _hosted jupyterlab: Hosted JupyterLab +++++++++++++++++++ There are many providers of hosted :ref:`JupyterLab` and :ref:`Jupyter Notebook`; where they run Jupyter in a shell or a VM on their servers for you and you connect over your internet connection. - https://github.com/markusschanta/awesome-jupyter#hosted-notebook-solutions - :ref:`Cocalc` - :ref:`Google AI Platform` has hosted JupyterLab Notebooks. - :ref:`Google Colab` is a fork of :ref:`Jupyter Notebook`. - :ref:`GitHub Codespaces` (:ref:`VSCode`) - :ref:`ml-workspace` -- https://github.com/ml-tooling/ml-workspace (:ref:`VSCode`) - :ref:`MyBinder.org` .. index:: JupyterHub .. _jupyterhub: JupyterHub ++++++++++++ | Src: https://github.com/jupyter/jupyterhub | Docs: https://jupyterhub.readthedocs.io/en/latest/ | Docs: https://github.com/jupyter/jupyterhub/wiki | Docs: https://github.com/jupyter/jupyterhub/wiki/Authenticators | Docs: https://github.com/jupyter/jupyterhub/wiki/Spawners JupyterHub makes it easy to serve :ref:`Jupyter Notebook` and/or :ref:`Jupyter Lab` for multiple users on one or more servers. - JupyterHub spawns individual Jupyter Notebook / JupyterLab server instances for logged-in users. - JupyterHub enables users to log-in with Authenticator backends: system users, LDAP, SSO, OAuth (e.g. Google accounts) - If so configured, JupyterHub can launch additional servers to serve one or more Notebook/Lab :ref:`Docker` containers and then shut those down when they're idle or, for example, when a course session is complete. .. index:: nbconvert .. _nbconvert: nbconvert +++++++++++ | Src: https://github.com/jupyter/nbconvert | Docs: https://nbconvert.readthedocs.io/en/latest/ nbconvert is the code that converts (transforms) an ``.ipynb`` notebook (:ref:`nbformat` :ref:`JSON `) file into an output representation (e.g. :ref:`HTML`, HTML slides (:ref:`reveal.js`), :ref:`LaTeX`, PDF, ePub, Mobi). * nbconvert is included with :ref:`Jupyter Notebook` and :ref:`JupyterLab` .. code:: bash pip install nbconvert # pip install -e git+https://github.com/jupyter/nbconvert@master#egg=nbconvert jupyter nbconvert --to html mynotebook.ipynb .. index:: reveal.js .. _reveal.js: ---------- reveal.js ---------- | Src: https://github.com/hakimel/reveal.js reveal.js is a :ref:`Javascript` and :ref:`HTML` library for slide presentations served from an HTML file. - Reveal.js slides can be in a 1-dimensional or a 2-dimensional arrangement. - You can generate reveal.js slides from Jupyter notebooks in two ways: with ``nbconvert --to slides`` or with the GUI: "File" > "Export Notebok As..." > "Export Notebook to reveal.js slides" .. code:: bash jupyter nbconvert --to slides mynotebook.ipynb .. note:: Presentation content that doesn't fit on a slide is hidden and unscrollable: *only put a slide worth of data in each cell for a Jupyter reveal.js presentation*. Alternatives to presenting notebooks as reveal.js slides: - Increase the browser font size (Jupyter Notebook) - "View" > "Presentation Mode" (JupyterLab) - Select a keyboard shortcut set use the "Select Cell Below" / "Select Cell Above" keyboard shortcuts to highlight cells and scroll them into view - Press "" - Press "j" to "Select Cell Below" - Press "k" to "Select Cell Above" - The :ref:`RISE` extension also generates reveal.js slides. .. index:: RISE .. _rise: ------ RISE ------ | Src: https://github.com/damianavila/RISE | Docs: https://rise.readthedocs.io/en/latest/ RISE is a :ref:`Jupyter Notebook` and :ref:`JupyterLab` extension that generates live :ref:`reveal.js` presentations from Jupyter notebooks. - Install the RISE extension - Click the RISE button to generate a *live* :ref:`reveal.js` slide presentation wherein you can execute cells on the slides with "Ctrl-Enter" and "Shift-Enter" just like you can in the Notebook interface. .. index:: nbformat .. _nbformat: nbformat ++++++++++ | Src: https://github.com/jupyter/nbformat | Docs: https://nbformat.readthedocs.io/en/latest/ | Docs: https://nbformat.readthedocs.io/en/latest/format_description.html#backward-compatible-changes The :ref:`Jupyter notebook` (``.ipynb``) format is a versioned :ref:`JSON ` format for storing metadata and input/output sequences. Usually, when the nbformat changes, notebooks are silently upgraded to the new version on the next save. .. note:: nbformat v3 and above add a **kernelspec** attribute to the nbformat :ref:`JSON `, because ``.ipynb`` files can now contain code for languages other than :ref:`Python`. * nbformat does not specify any schema for the user-supplied metadata dict (TODO: :ref:`nbmeta`), so JSON that conforms to an externally managed :ref:`JSON-LD ` ``@context`` would work. .. index:: nbgrader .. _nbgrader: nbgrader ++++++++ | Src: https://github.com/jupyter/nbgrader | Docs: https://nbgrader.readthedocs.io/en/stable/ | Docs: https://nbgrader.readthedocs.io/en/stable/user_guide/ | Docs: https://nbgrader.readthedocs.io/en/latest/ | Docs: https://nbgrader.readthedocs.io/en/latest/user_guide/ | Docs: https://nbgrader.readthedocs.io/en/latest/command_line_tools/ nbgrader is a solution for centrally receiving and grading :ref:`Jupyter notebooks `. - You mark notebook cells as TODO See also: * :ref:`CoCalc` Course management * https://wrdrd.github.io/docs/consulting/education-technology#jupyter-and-tdd * :ref:`jupyter and tdd` .. index:: nbviewer .. _nbviewer: nbviewer +++++++++++ | Homepage: https://nbviewer.jupyter.org | Src: git https://github.com/jupyter/nbviewer | Dockerfile: https://github.com/jupyter/nbviewer/blob/master/Dockerfile (``nbviewer``) is an application for serving read-only versions of Jupyter notebooks from HTTP URLs. - When you enter a URL, GitHub ``username``, GitHub ``username/repo``, or Gist ID into the text box at https://nbviewer.jupyter.org/ and click 'Go!' (or press Enter), nbviewer nbconverts the notebook to HTML or shows a file browser and branch/tag selector for the git repo. - You do not need to look up the raw GitHub URL for the notebook, because nbviewer automatically rewrites the GitHub /blob/ file URL to a raw.githubusercontent.com URL. - GitHub now also renders static ``.ipynb`` files, CSV, SVG, and PDF. However, GitHub does not execute any JS in the notebook due to security concerns (XSS) - GitLab renders Jupyter notebooks with JS. .. index:: runipy .. _runipy: runipy +++++++ | Src: git https://github.com/paulgb/runipy | PyPI: https://pypi.python.org/pypi/runipy runipy runs :ref:`Jupyter notebooks ` from a :ref:`Shell ` commandline, generates `HTML` reports, and can write errors to stderr. :ref:`Jupyter notebook ` *manual* test review process: .. code:: # - run Jupyter Notebook server !jupyter notebook # - Browser # - navigate to / upload / drag and drop the notebook !web http://localhost:8888 # or https:// # - (optional) click 'TODO Restart Kernel' # - (optional) click 'Cell' > 'All Output' > 'Clear' # - click 'Cell' > 'Run All' # - [wait] # - visually seek for the first ERRoring cell (scroll) # - review the notebook for (i, o) in notebook_cells: human.manually_review((i, o)) # - Compare the files on disk with the most recent commit (HEAD) !git status && git diff !git diff mynotebook.ipynb # - Commit the changes !git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb :ref:`Jupyter notebook ` TODO review process: .. code:: # - run Jupyter Notebook server !jupyter notebook # - Browser # - navigate to / upload / drag and drop the notebook !web http://localhost:8888 # or https:// # - (optional) click 'TODO Restart Kernel' # - (optional) click 'Cell' > 'All Output' > 'Clear' # - click 'Cell' > 'Run All' # - [wait] # - visually seek for the first ERRoring cell (scroll) # - review the notebook for (i, o) in notebook_cells: human.manually_review((i, o)) # - Compare the files on disk with the most recent commit (HEAD) !git status && git diff !git diff mynotebook.ipynb # - Commit the changes !git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb :ref:`Jupyter notebook ` runipy review process: .. code:: # - runipy the Jupyter notebook !runipy mynotebook.ipynb # - review stdout and stderr from runipy # - review in browser (optional; recommended) # - navigate to the converted HTML !web ./mynotebook.ipynb.html # - visually seek for the first WEEoring cell (scroll) # - review the notebook for (i, o) in notebook_cells: human.manually_review((i, o)) # - Compare the files on disk with the most recent commit (HEAD) !git status && git diff !git diff mynotebook.ipynb* # - Commit the changes !git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb* * An example of runipy usage in a :ref:`Makefile `: https://github.com/westurner/notebooks/blob/gh-pages/Makefile .. index:: jupyter_contrib_nbextensions .. _jupyter_contrib_nbextensions: jupyter_contrib_nbextensions ++++++++++++++++++++++++++++++ | Src: https://github.com/ipython-contrib/jupyter_contrib_nbextensions | Docs: https://github.com/ipython-contrib/jupyter_contrib_nbextensions#1-install-the-python-package | CondaPkg: ``jupyter_contrib_nbextensions`` - https://github.com/ipython-contrib/jupyter_contrib_nbextensions#pip - https://github.com/ipython-contrib/jupyter_contrib_nbextensions#conda .. index:: NBPresent .. _nbpresent: NBPresent ++++++++++++ | Src: https://github.com/Anaconda-Platform/nbpresent | Docs: https://docs.continuum.io/anaconda/jupyter-notebook-extensions#notebook-present remix your :ref:`Jupyter Notebooks ` as interactive slideshows .. index:: Anaconda Jupyter Notebook Extensions .. _anaconda jupyter notebook extensions: Anaconda Jupyter Notebook Extensions +++++++++++++++++++++++++++++++++++++++ | Src: https://github.com/Anaconda-Platform/anaconda-nb-extensions | Docs: https://docs.continuum.io/anaconda/jupyter-notebook-extensions * :ref:`Conda` environments, :ref:`Anaconda`, :ref:`Jupyter Notebook` .. index:: CoCalc .. index:: SageMathCloud .. _cocalc: .. _sagemathcloud: CoCalc ~~~~~~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/CoCalc | Homepage: https://cocalc.com/ | Src: https://github.com/sagemathinc/cocalc | Src: https://github.com/sagemathinc/cocalc-docker | ChromeExt: https://chrome.google.com/webstore/detail/cocalc/eocdndagganmilahaiclppjigemcinmb | Docs: https://doc.cocalc.com/ | Docs: https://doc.cocalc.com/teaching-instructors.html | Docs: https://doc.cocalc.com/teaching-students.html * https://cocalc.com/ * Interactive Worksheets * Course Management: Assignments, Handouts, :ref:`nbgrader` * :ref:`LaTeX` Editor * :ref:`Jupyter Notebook` * Linux Terminal * :ref:`Sage Math` .. index:: Google Colab .. index:: Colab .. _colab: .. _google colab: Google Colab ~~~~~~~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Project_Jupyter#Colaboratory | Homepage: https://colab.research.google.com/ | Src: https://github.com/googlecolab | Docs: https://research.google.com/colaboratory/faq.html | Awesome: https://github.com/firmai/awesome-google-colab Google Colab is a hosted :ref:`Jupyter Notebook` system. - Colab has a number of packages installed in the default environment. If you want additional packages, you need to ``!pip install`` them once when you first open the notebook. - Colab is forked from a previous version of Jupyter Notebook, and so does not have some newer Jupyter Notebook or any Jupyter Lab features. - :ref:`ipywidgets` are not yet implemented on Colab. - Colab saves to Google Drive. - Colab instances are free and can use some GPU time if needed. - There is a Colab Pro. - Google AI Platform Notebooks hosts :ref:`JupyterLab` notebooks: https://cloud.google.com/ai-platform-notebooks .. index:: Dotfiles .. _dotfiles: Dotfiles ========== Dotfiles are userspace :ref:`shell ` configuration in files that are often prefixed with "dot" (e.g. ``~/.bashrc`` for :ref:`Bash`) * https://dotfiles.github.io/ * westurner/dotfiles | Src: git https://github.com/westurner/dotfiles | Src: https://github.com/westurner/dotfiles/blob/master/etc/.bashrc | Docs: https://westurner.github.io/dotfiles/ | Docs: https://westurner.github.io/dotfiles/usage#bash Features + [x] Linear ``etc/bash`` and ``etc/zsh`` (``/etc/bash``) load sequence + [x] :ref:`make` ``Makefile`` to log the whole load sequence and grep out docs + [x] :ref:`HubFlow` :ref:`git` branches + [x] :ref:`venv`, :ref:`virtualenv`, :ref:`virtualenvwrapper` + [x] oh-my-zsh + [-] bash-it .. index:: Dotvim .. _dotvim: Dotvim ~~~~~~~~ Dotvim is a conjunction / contraction of :ref:`Dotfiles` and :ref:`Vim` (in reference to a ``~/.vim/`` directory and/or a ``~/.vimrc``). | Src: git https://github.com/westurner/dotvim | Src: https://github.com/westurner/dotvim/blob/master/vimrc | Src: https://github.com/westurner/dotvim/blob/master/vimrc.full.bundles.vimrc | Src: https://github.com/westurner/dotvim/blob/master/vimrc.tinyvim.bundles.vimrc | Docs: https://westurner.github.io/dotfiles/usage#vim .. index:: Venv .. _venv: Venv ~~~~~ | Docs: https://westurner.github.io/dotfiles/venv | Docs: https://westurner.github.io/dotfiles/dotfiles.venv | Src: https://github.com/westurner/dotfiles/blob/develop/src/dotfiles/venv/ | Src: https://github.com/westurner/dotfiles/blob/develop/etc/bash/10-bashrc.venv.sh Venv is a tool for making working with :ref:`Virtualenv`, :ref:`Virtualenvwrapper`, :ref:`Bash`, :ref:`ZSH`, :ref:`Vim`, and :ref:`IPython` within a project context very easy. Venv defines standard :ref:`fhs` and :ref:`Python` paths, environment variables, and aliases for routinizing workflow. .. table:: Venv paths and cdaliases :class: table-striped table-responsive +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | **var name** | **description** | **cdaliases** | **example path** | | | | | | | | | Bash: ``cdhelp`` | | | | | | | | | | IPython: ``%cdhelp`` | | | | | | | | | | Vim: ``:Cdhelp`` | | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``HOME`` | user home directory | Bash/ZSH: ``cdh``, ``cdhome`` | ``~/`` | | | | | | | | | IPython: ``%cdh``, ``%cdhome`` | | | | | | | | | | Vim: ``:Cdh``, ``:Cdhome`` | | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``__WRK`` | workspace root | ``cdwrk`` (ibid.) | ~/``-wrk`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``WORKON_HOME`` | virtualenvs root | ``cdwh``, ``cdworkonhome``, ``cdve`` | ~/-wrk/``-ve27`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``CONDA_ENVS_PATH`` | condaenvs root | ``cdch``, ``cdcondahome`` | ~/-wrk/``-ce27`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``VIRTUAL_ENV`` | virtualenv root | ``cdv``, ``cdvirtualenv`` | ~/-wrk/-ve27/``dotfiles`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_BIN`` | virtualenv executables | ``cdb``, ``cdbin`` | ~/-wrk/-ve27/dotfiles/``bin`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_ETC`` | virtualenv configuration | ``cd``, ``cdetc`` | ~/-wrk/-ve27/dotfiles/``etc`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_LIB`` | virtualenv lib directory | ``cdl``, ``cdlib`` | ~/-wrk/-ve27/dotfiles/``lib`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_LOG`` | virtualenv log directory | ``cdlog`` | ~/-wrk/-ve27/dotfiles/``var/log`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_SRC`` | virtualenv source repositories | ``cds``, ``cdsrc`` | ~/-wrk/-ve27/dotfiles/``src`` | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ | ``_WRD`` | virtualenv working directory | ``cdw``, ``cdwrd`` | ~/-wrk/-ve27/dotfiles/``src/dotfiles`` | | | | | | +---------------------+--------------------------------+--------------------------------------+----------------------------------------+ To generate this venv config: .. code:: bash python -m dotfiles.venv.ipython_config --print-bash dotfiles venv.py --print-bash dotfiles venv --print-bash dotfiles docs venv --print-bash dotfiles ~/path venv --print-bash ~/-wrk/-ve27/dotfiles ~/path To generate a default venv config with a prefix of ``/``: .. code:: bash venv --print-bash --prefix=/ To launch an interactive shell within a venv: .. code:: bash venv --run-bash dotfiles venv -xb dotfiles .. note:: ``pyvenv`` is the :ref:`Virtualenv` -like functionality now included in :ref:`Python >= 3.3 ` (``python3 -m venv``) Python pyvenv docs: https://docs.python.org/3/library/venv.html .. index:: Virtualenv .. _virtualenv: Virtualenv ~~~~~~~~~~~~~~~~~~~~ | Homepage: https://virtualenv.pypa.io/en/latest/ | Docs: https://virtualenv.pypa.io/en/latest/ | Src: git https://github.com/pypa/virtualenv | PyPI: https://pypi.python.org/pypi/virtualenv | IRC: #pip Virtualenv is a tool for creating reproducible :ref:`Python` environments. Virtualenv sets the shell environment variable ``$VIRTUAL_ENV`` when active. Virtualenv installs a copy of :ref:`Python`, :ref:`Setuptools`, and :ref:`Pip` when a new virtualenv is created. A virtualenv is activated by ``source``-ing ``${VIRTUAL_ENV}/bin/activate``. Paths within a virtualenv are more-or-less :ref:`FHS ` standard paths, which makes virtualenv structure very useful for building chroot and container overlays. A standard virtual environment:: bin/ # pip, easy_install, console_scripts bin/activate # source bin/activate to work on a virtualenv include/ # (symlinks to) dev headers (python-dev/python-devel) lib/ # libraries lib/python2.7/distutils/ lib/python2.7/site-packages/ # pip and easy_installed packages local/ # symlinks to bin, include, and lib src/ # editable requirements (source repositories) # also useful etc/ # configuration var/log # logs var/run # sockets, PID files tmp/ # mkstemp temporary files with permission bits srv/ # local data :ref:`Virtualenvwrapper` wraps virtualenv. .. code-block:: bash echo $PATH; echo $VIRTUAL_ENV python -m site; pip list virtualenv example # mkvirtualenv example source ./example/bin/activate # workon example echo $PATH; echo $VIRTUAL_ENV python -m site; pip list ls -altr $VIRTUAL_ENV/lib/python*/site-packages/** # lssitepackages -altr .. note:: :ref:`Venv` extends :ref:`virtualenv` and :ref:`virtualenvwrapper`. .. note:: Python 3.3+ now also contain a script called **venv**, which performs the same functions and works similarly to virtualenv: ``_. .. index:: Virtualenvwrapper .. _virtualenvwrapper: Virtualenvwrapper ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Docs: https://virtualenvwrapper.readthedocs.io/en/latest/ | Src: hg https://bitbucket.org/dhellmann/virtualenvwrapper | PyPI: https://pypi.python.org/pypi/virtualenvwrapper Virtualenvwrapper is a tool which extends virtualenvwrapper. Virtualenvwrapper provides a number of useful shell commands and python functions for working with and within :ref:`virtualenvs `, as well as project event scripts (e.g. ``postactivate``, ``postmkvirtualenv``) and two filesystem configuration variables useful for structuring development projects of any language within :ref:`virtualenvs `: ``$PROJECT_HOME`` and ``$WORKON_HOME``. Virtualenvwrapper is sourced into the shell:: # pip install --user --upgrade virtualenvwrapper source ~/.local/bin/virtualenvwrapper.sh # sudo apt-get install virtualenvwrapper source /etc/bash_completion.d/virtualenvwrapper .. note:: :ref:`Venv` extends :ref:`virtualenv` and :ref:`virtualenvwrapper`. .. code-block:: bash echo $PROJECT_HOME; echo ~/workspace # venv: ~/-wrk cd $PROJECT_HOME # venv: cdp; cdph echo $WORKON_HOME; echo ~/.virtualenvs # venv: ~/-wrk/-ve27 cd $WORKON_HOME # venv: cdwh; cdwrk mkvirtualenv example workon example # venv: we example cdvirtualenv; cd $VIRTUAL_ENV # venv: cdv echo $VIRTUAL_ENV; echo ~/.virtualenvs/example # venv: ~/-wrk/-ve27/example mkdir src ; cd src/ # venv: cds; cd $_SRC pip install -e git+https://github.com/westurner/dotfiles#egg=dotfiles cd src/dotfiles; cd $VIRTUAL_ENV/src/dotfiles # venv: cdw; cds dotfiles head README.rst # venv: cdpylib cdsitepackages # venv: cdpysite lssitepackages deactivate rmvirtualenv example lsvirtualenvs; ls -d $WORKON_HOME # venv: lsve; lsve 'ls -d' .. index:: Window Managers .. _window managers: Window Managers ================ | Wikipedia: https://en.wikipedia.org/wiki/Window_manager | Docs: https://wiki.archlinux.org/index.php/Window_manager .. index:: Compiz .. _compiz: Compiz ~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Compiz | Homepage: https://launchpad.net/compiz | Docs: http://wiki.compiz.org/ | Src: bzr branch lp:compiz Compiz is a window compositing layer for :ref:`X11` which adds lots of cool and productivity-enhancing visual capabilities. Compiz works with :ref:`Gnome`, :ref:`KDE`, and :ref:`Qt` applications. .. index:: f.lux .. _f.lux: f.lux ~~~~~~ | Homepage: https://justgetflux.com/ | Download: https://justgetflux.com/dlmac.html | Download: https://justgetflux.com/dlwin.html | Src: git https://github.com/Kilian/f.lux-indicator-applet | Docs: https://justgetflux.com/linux.html | Docs: https://justgetflux.com/ios.html | Docs: https://justgetflux.com/research.html f.lux is a userspace utility for gradually adjusting the blue color channel throughout the day; or as needed. * A similar effect can be accomplished with the :ref:`X11` ``xgamma`` command (e.g. for :ref:`Linux` platforms where the latest f.lux is not yet available). A few keybindings from an :ref:`i3wm` configuration `here `_: .. code:: # [...] #L105 set $xgamma_reset xgamma -gamma 1.0 set $xgamma_soft xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9 set $xgamma_soft_red xgamma -bgamma 0.4 -ggamma 0.6 -rgamma 0.9 # [...] #L200 ## Start, stop, and reset xflux # [ -- start xflux bindsym $mod+bracketleft exec --no-startup-id $xflux_start # ] -- stop xflux bindsym $mod+bracketright exec --no-startup-id $xflux_stop # ] -- reset gamma to 1.0 bindsym $mod+Shift+bracketright exec --no-startup-id $xgamma_reset # [ -- xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9 bindsym $mod+Shift+bracketleft exec --no-startup-id $xgamma_soft # \ -- xgamma -bgamma -0.4 -ggamma 0.4 -rgamma 0.9 bindsym $mod+Shift+p exec --no-startup-id $xgamma_soft_red .. index:: Gnome .. _gnome: Gnome ~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/GNOME | Homepage: https://www.gnome.org/ | Docs: https://help.gnome.org/ | Src: https://git.gnome.org/browse/ * https://wiki.gnome.org/GnomeLove .. index:: i3wm .. _i3wm: i3wm ~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://i3wm.org/ | Download: https://i3wm.org/downloads/ | Docs: https://i3wm.org/docs/ | Src: git git://code.i3wm.org/i3 i3wm is a tiling window manager for :ref:`X11` (:ref:`Linux`) with extremely-configurable :ref:`Vim`-like keyboard shortcuts. i3wm works with :ref:`Gnome`, :ref:`KDE`, and :ref:`Qt` applications. * An example open source i3wm ``i3/config`` :ref:`dotfile `: https://github.com/westurner/dotfiles/blob/master/etc/i3/config .. index:: KDE .. _kde: KDE ~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/KDE | Homepage: https://www.kde.org/ | Docs: https://docs.kde.org/ | Docs: https://www.kde.org/documentation/ | Src: https://techbase.kde.org/Getting_Started/Sources | Src: https://techbase.kde.org/Getting_Started/Sources/Subversion | Src: https://techbase.kde.org/Development/Git | Src: https://projects.kde.org/projects KDE is a GUI framework built on :ref:`Qt`. KWin is the main KDE window manager for :ref:`X11`. .. index:: Qt .. _qt: Qt ~~~ | Wikipedia: https://en.wikipedia.org/wiki/Qt_Project | Homepage: https://qt-project.org/ | Homepage: https://www.qt.io/ | Docs: https://doc.qt.io/ | Docs: https://doc.qt.io/qt-5/qtexamplesandtutorials.html | Docs: https://www.qt.io/contribute/ | Docs: https://wiki.qt.io/Main_Page | Docs: https://wiki.qt.io/Get_the_Source | Src: git https://code.qt.io/cgit/ Qt is a Graphical User Interface toolkit for developing applications with Android, iOS, :ref:`OSX`, Windows, Embedded :ref:`Linux`, and :ref:`X11`. .. index:: Wayland .. _wayland: Wayland ~~~~~~~~~ | Wikipedia: ``_ | Homepage: https://wayland.freedesktop.org/ | Src: https://gitlab.freedesktop.org/wayland/wayland Wayland is a display server protocol for GUI window management. Wayland is an alternative to :ref:`X11` servers like XFree86 and X.org. The reference Wayland implementation, Weston, is written in :ref:`C`. .. index:: X Window System .. index:: X11 .. _x11: X11 ~~~~ | Wikipedia: https://en.wikipedia.org/wiki/X_Window_System | Homepage: https://www.x.org/ | Docs: https://www.x.org/wiki/Documentation/ | Src: git git://anongit.freedesktop.org/git/xorg/ X Window System (X, X11) is a display server protocol for window management (drawing windows on the screen). Most UNIX and :ref:`Linux` systems utilize XFree86 or the newer X.org X11 window managers. :ref:`Gnome`, :ref:`KDE`, :ref:`I3wm`, :ref:`OSX`, and :ref:`Compiz` build upon X11. .. index:: Browsers .. _browsers: Browsers ========== .. index:: Blink .. _blink: Blink ~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Blink_(layout_engine)>`__ | Homepage: https://www.chromium.org/blink | Src: https://src.chromium.org/viewvc/blink/trunk/ | Src: git https://chromium.googlesource.com/chromium/blink/ | Docs: https://www.chromium.org/blink#TOC-Subpage-Listing | Docs: https://www.chromium.org/blink/developer-faq Blink is a :ref:`web browser ` layout engine written in :ref:`C++` which was forked from :ref:`WebKit`. * Blink now powers :ref:`Chrome` and :ref:`Chromium` (Desktop, Mobile), :ref:`Opera`, Amazon Silk, :ref:`Android` WebView 4.4+, and :ref:`Qt` WebEngine .. index:: Chrome .. _chrome: Chrome ~~~~~~~ | Wikipedia: https://en.wikipedia.org/wiki/Google_Chrome | Homepage: https://www.google.com/chrome/ | Download: https://www.google.com/chrome/browser/desktop/ | Download: https://www.google.com/chrome/browser/mobile/ | AndroidApp: https://play.google.com/store/apps/details?id=com.android.chrome | AndroidAppBeta: https://play.google.com/store/apps/details?id=com.chrome.beta | iOSApp: https://itunes.apple.com/us/app/chrome-web-browser-by-google/id535886823 | Docs: https://developer.chrome.com | Docs: https://developer.chrome.com/extensions/devguide Google Chrome is a Web Browser built from the open source :ref:`Chromium` browser. * Google Chrome is now based on :ref:`Blink`. * Google Chrome was based on :ref:`WebKit`. * Google Chrome includes and updates Adobe Flash, pdf.js See also: :ref:`ChromeOS`. .. index:: Chromium .. _chromium: Chromium ~~~~~~~~~~ | Wikipedia: ``__ | Homepage: https://www.chromium.org/Home | Src: git https://chromium.googlesource.com/chromium/src | Src: https://chromium.googlesource.com/chromium/src.git | Docs: https://www.chromium.org/developers | Docs: https://www.chromium.org/developers/coding-style The Chromium Projects include the :ref:`Chromium` Browser and :ref:`chromiumos`. * Chromium Projects are written in :ref:`C++11 `, :ref:`C`, IDL, :ref:`Jinja2`, :ref:`Python`, :ref:`Javascript`, :ref:`HTML-`, and :ref:`CSS-`. * :ref:`Chrome` and :ref:`ChromeOS` build from :ref:`Chromium` project sources. .. index:: Chrome DevTools .. _chrome-devtools: Chrome DevTools ++++++++++++++++ | Homepage: https://developer.chrome.com/devtools | Docs: https://developer.chrome.com/devtools How to open Chrome (and Firefox) DevTools: * Right-click > "Inspect Element" * Linux: ``i`` * OSX: ``