I know that I can install Jython with Java and that I can use Jython where I use Python. The Jython shell is working fine.

In Jython, how can I install libraries like lxml, Scrappy and BeautifulSoup that I’d normally install via pip or easy_install

Some Python modules, like lxml, have required components in C. These won’t work in Jython.

Most Python packages will work fine, and you can install them using the same tools as you use in CPython. This is described in Appendix A of Jython Book:

To get setuptools, download ez_setup.py from
http://peak.telecommunity.com/dist/ez_setup.py. Then, go to the
directory where you left the downloaded file and execute:

$ jython ez_setup.py

[The easy_install script will be] installed to the bin directory of
the Jython installation (/home/lsoto/jython2.5.0/bin in the example
above). If you work frequently with Jython, it’s a good idea to
prepend this directory to the PATH environment variable, so you don’t
have to type the whole path each time you want to use easy_install or
other scripts installed to this directory.

Testing it myself, after installing setuptools in Jython, pip installed correctly:

$ sudo /usr/bin/jython2.5.2b1/bin/easy_install pip
Searching for pip
Installing pip-2.5 script to /usr/bin/jython2.5.2b1/bin
Installing pip script to /usr/bin/jython2.5.2b1/bin

Installed /usr/bin/jython2.5.2b1/Lib/site-packages/pip-1.0.2-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip

$ sudo /usr/bin/jython2.5.2b1/bin/pip install bottle
Downloading/unpacking bottle
  Downloading bottle-0.9.6.tar.gz (45Kb): 45Kb downloaded
  Running setup.py egg_info for package bottle
    Installing collected packages: bottle
  Running setup.py install for bottle
Successfully installed bottle
Cleaning up...

$ jython
Jython 2.5.2b1 (Release_2_5_2beta1:7075, Jun 28 2010, 07:44:20) 
[Java HotSpot(TM) 64-Bit Server VM (Apple Inc.)] on java1.6.0_26
Type "help", "copyright", "credits" or "license" for more information.
>>> import bottle
>>> bottle
<module 'bottle' from '/usr/bin/jython2.5.2b1/Lib/site-packages/bottle$py.class'>

As of v2.7b4, the Jython distribution includes the ensurepip module, which simplifies installation of pip and setuptools:

jython -m ensurepip

Beware of sys.platform==’win32′ issue that will get in your way of using PyPI packages which relying on this method to determine host platform.

Being Jython though you have the power of the Java libraries, not the limitation of being unable to install a few python C libraries.

For example you would be better using Jsoup instead of Beautiful soup or go for a full solution like Jtidy.

Use Jaxp instead of lxml.

Also another option that fits your reuirements is NekoHTML

I have both CPython and Jython installed. Here is what I do if I want to install a package in Jython via pip.

jython -m pip install <package_name>

For example to install the robot framework in jython (as I want to write the keyword libraries in Java), I did

jython -m pip install robotframework