Linux HowTo: Library Files

Almost all of Linux programs depend on some thing called library files. 

-the idea behind library files:

            . to simplify programmers lives by providing a repository for commonly used shared program fragments.

            . this reduces the main program size as you only need to invoke the libraries as a reference link (eg #include xyz.so)

            . shared library files usually have a '.so' filename extension; '.so' = shared object

-note: besides shared libraries, there are static libraries which have a '.a' filename extn. they are used for backward compatibility to older programs.

-eg:         libc.so              - the library for C programs.

              Gimp toolkit      - the library for onscreen widgets (buttons, scrollbars, menubars etc) in Gnome

              Qt                     - the library for onscreen widgets (buttons, scrollbars, menubars etc) in KDE

-the library path for the system is set in the file /etc/ld.so.conf or in the env variables LIBPATH or LD_LIBRARY_PATH

-displaying libraries that a program depends upon:

-eg:         # ldd  /bin/ls       [ shows the library dependencies for the /bin/ls binary ]

-display config info for libraries, currently loaded in memory:

              # ldconfig

Linux HowTo: rpm, yum, dpkg, apt

First the Basics:

-libraries:

             . os software modules that can be used by many different programs.

-packages:

             . software files that are installed on a linux system.

             . usually distributed as tarballs in two forms - rpm or debian.

-package info, mainly dependencies on other s/w is tracked in the form of a database.

-Although it is possible to install both forms of packages on a system (ie both rpm and debian), it is recommended to go with only one.

-rpm is the most popular package form for linux. redhat developed rpm and released it under gpl.

-redhat linux is available for atleast 5 cpu architectures:

              . x86                  - first by intel/then by amd etc

              . x86-64             - first by amd/then by intel etc

              . alpha               - dec/compaq/hp

              . ia-64                - intel itanium

              . sparc               - sun

-yellow dog linux is a distribution of linux developed by fixstars in 1999 for powerpc cpus used then by macos pcs.

-for the most part, the source rpms can be used to build packages on any cpu architecture.

-pkg name convention:

              . pkgname-a.b.c-x.arch.rpm                    

      -where a.b.c - version

                       x - build

                  arch - cpu type (most commonly i386, which represents x86 cpus from 80386 onwards)

-eg: . samba-3.0.25b-5mdv.x86_64.rpm

-useful rpm cmds:

             # rpm -q a i <rpm pkg>                 [ query ]

             # rpm -i v h <rpm pkg>                 [ U for upgrade - generally install is safer than upgrade ]

             # rpm --rebuild <rpm pkg>            [ rebuilds the binaries from the source code]

-source rpm files are actually modified cpio archives.

-so it is possible to extract individual files from the source rpm by reconverting them to cpio archives.

-for this the program called 'rpm2cpio' is used.

-eg:        # rpm2cpio blabla.src.rpm > blabla.cpio

-then      # cpio -i --make-directories < blabla.cpio

-or          # cpio -idmv < blabla.cpio

-this results in extraction of files in the current dir.

Yum cmd -yum originated with yellow dog linux but now incorporated with red hat.

-it automates a lot of tasks in searching, downloading and installing rpms.

-one drawback of yum is that it can only install what is in the repository and that too over the Internet.

-eg:        # yum list | install | upgrade | erase | info .. <pkg name>

-the main rpm config file is /usr/lib/rpm/rpmrc.

-the main yum config file is /etc/yum.conf ; other config files are in /etc/yum.repos.d dir.

-debian does not focus on flashy gui but its derivatives like ubuntu do. debian tries to be as opensource and bug free as possible.

-gnu hurd kernel != gnu linux kernel

-debian pkgs are incompatible with rpm, but the principles of operation are similar.

-dpkg cmd - cmd to manage debian pkgs ( like rpm cmd that manages rpm pkgs ).

-eg:        # dpkg -i r p .. <pkg name> ( i= install, r= remove, p= print info etc )

-dpkg can be useful for managing one or a few packages.

-apt-get cmd is the debian equivalent of yum in rpm world - it automates several steps in managing deb pkgs.

-apg-get looks for config info about pkg locations in /etc/apt/sources.list

-eg:        # apt-get check | install | upgrade | remove <pkg name>

-debian pkgs usually have '.deb' extn (like .rpm for rpm pkgs).

-the main config file for dpkg is '/etc/dpkg/dpkg.cfg'.

-the main config file for apt are '/etc/apt/sources.list' and '/etc/apt/apt.conf'.

-package conversion:

             . sometimes it requires a pkg conversion between rpm to deb or vice versa.

             . utilities like 'alien' help to do that.

-alien requires that both pkg managers be installed (rpm and dpkg).

-eg:       # alien --to-rpm <aprog.deb>

             # alien --to-deb <aprog.rpm>

-common cause of package dependency problems:

             . missing libraries

             . incompatible libraries

             . duplicate / mismatched names

-common answers to package dependency problems:

             . force the install

             . rebuilding pkg from source

             . modifying system config files, etc..

Linux HowTo: dd command

The dd command is an old command carried over from the Mainframe computing days of the yore. It stands for 'Data Dump'. Also known as 'data destroyer' as a pun, in case not used carefully (since it can destroy disk partitioning very innately and cleanly). It is considered as a good choice of disk to disk backup methods at a very native level, it can even be used to copy whole partitions or filesystems. For details refer below:

-dd backup method:

           . sequential file image of a partition irrespective of the filesystem knowledge that reside on the partition. dd cmd does so.

           . backup of partition using only the used fs. dump cmd does so.

-dd requires that the restore be done on an equal or bigger partition.

-dump can restore to partition that is sufficient to hold the used fs.

-dd cmd syntax:

           # dd if=<input file path> of=<output file name>

-eg:     # dd if=/dev/sda3 of=backup-1                     [ copies /dev/sda3 partition to a file called 'backup-1' in pwd ]

           # dd if=/dev/sda3 of=/dev/fd0                      [ copies /dev/sda3 partition to floppy drive ]

           # dd if=/dev/sda2 | gzip > backup-2            [ copies /dev/sda2 partition as a gzipped file 'backup-2' in pwd ]

-note:

          . dd cmd copies all blocks - whether used or unused (ie empty); also the output location needs to be atleast as big as the input.

          . dd cmd can be used an easy way to troubleshoot IO:

           # time dd if=/u03/oradata/devenv/one.dbf of=/dev/null                 [ reads file and times it ] 

           # dd --help                                                   [ or man dd for more help/details ]

Internet Kill Switch: A not good Idea

Did you hear that news about the Internet Kill Switch Bill? Well the idea is that in the unfortunate event of an Terrorist or Cyber Attack on the US, the White House can turn off certain parts or most of the Internet from being accessible to anyone. It is basically an over-arching security measure against cyber attacks. There is much debate about whether this is justified and/or even possible. But nonetheless, it is an area that I think every one of us who access the Internet should be aware of.
However, the collateral damage is that the very concept of Internet as we know is something that can be turned off from the netizens as a result of this legislation.
The Internet evolved in the labs of government funded Defense labs in late 60s. But it quickly grew out of those realms from not just control perspective but also from applications perspective. Today the Internet is the worlds biggest network. It has all the information that you want and more. And its continuously evolving. They say the IPv4 protocol that can address about 4Billion addresses is not enough. So we have IPv6 which can address 10^38 individual nodes. And it is available in every part of the globe. Also to note, it has users and applications from all parts and corners of human imagination. Nutshell, you can't imagine today's world without the Internet. One of the fundamental reasons for the explosive growth of the Internet is that is not owned by any one single organization or government. It harbors individual development, Open collaboration of ideas and innovations from hundreds of thousands of users, user groups and orgs. Although some of the big organizations that have the direction of the Internet under tab are in the US like ICANN and IETF. But the point is the reach of the Internet is not limited to just US. Nor is its applicability. That said, the Internet was originally meant and still continues to be primarily a forum of Open Ideas, Collaboration and Freedom. It is the most Open Platform of our generation and must remain so for future ones. Therefore misuse of this extremely important medium by those who want to harm other humans or terrorize them or suppress freedom is wrong. Such usage of the Internet must be denounced and prevented with adequate security while maintaining the Internet access to millions others who just want to access Internet for Innovation and Collaboration for the good of all humans and generations.

Linux HowTo: My Top 10 Sys Admin Commands

Following is a the list of top 10 most useful system admin commands I find indispensable in managing my Linux Ubuntu Laptop.


1- Finding my ID:
        # id -a                                        - shows who I am, what groups I belong

2- What machine:
        # uname -a                                 - shows my OS version, kernel, Platform

3- OS Version (specific to Ubuntu):
        # lsb_release -a                          - shows my Ubuntu version and release

4- Hostname:
        # hostname                                 - shows me the hostname, allows to set new hostname also.

5- Run level:
        # who -r                                     - shows current and previous runlevel, default is 2 on Ubuntu

6- Initialization state:
        # init <n>                                    - sets the initialization state, n=0 is halt, n=6 is reboot, n=2 is default.
7- Date and Time:
        # date                                         - show system date and time. Also allows to set new date/time as root


8- Network Interfaces:     
        # ifconfig -a                                - shows all my network interfaces and status thereof

9- Network connectivity:
        # ping   <hostname>                   - polls a remote node and verifies network connectivity using ICMP

10- Network connectivity:
        # traceroute <hostname>            - Traces each hop in the path of a network packet

Linux HowTo: LDAP Basics

LDAP or Lightweight Directory Access Protocol
        - a set of open protocols developed at University of Michigan in 1992
        - it is a simple database of information on a network (not rdbms); microsoft active directory uses ldap concept.
        - data is organized in a hierarchial client server structure (like dns) - called 'directory info tree'   

-how ldap works?
        - client connects to ldap server    - this is called binding
        - server authenticates
        - client queries the server

-ldap can store a variety of data - like plain text, binary, images or public key certificates etc.
-it can serve as a authentication or id management system;       it can completely replace nis.
-it can serve as dns, mail router etc.

-ldap terms:
        . entry        -    unit of ldap data                -like primary key
        . attribute    -    addtional support data for entry        -like columns depending on primary key
        . objectclass    -    special attributes                -like composite primary keys
        . schema    -    a set of ldap rules                -like data dictionary
        . ldif        -    plaintext 'ldap data interchange format    -like sql output or sqlldr

-ldap daemons:
        . slapd    -    standalone ldap daemon        - like a listener
        . slurpd    -    standalone replication daemon    - like db link for data exchange bet servers
    -note:      openldap    -    is opensource implementation of ldap

-To install openldap server and clients in one go:
        # yum    -y install    openldap-servers    openldap-clients               [ multiple pkgs in one cmd ]
-To configure openldap server , edit the following file:
        # vim    /etc/openldap/slapd.conf
-To start / stop openldap server:
        # service  ldap    start | stop | status
-To query ldap server, the foll cmd is used:
        # ldapsearch    <options>
-To configure ldap client:
        # system-config-authentication

Linux HowTo: File Permissions Basics

Every File on a Linux system has permissions for owner, group and world. These three govern who can access a file. Note that everything on a Linux System can be resolved to a File--eg files, directories, keyboard (input), output (screen), error etc.

-if you have only read permi and no exec permi on a dir, you can neither view nor access the contents of that dir.

-if you have only exec permi and no read permi on a dir, you cannot view the contents of the dir but access them.

-Normally, programs run with the invokers permissions, not the owners.

-But with suid and sgid, the programs run with the owners permissions, not the invokers.

-suid and sgid have an 's' bit in place of the 'x' bit in the permission list.

-such programs are called suid programs or sgid programs.

-octal value of suid = 4, sgid =2 and suid+sgid =6. eg: chmod 6755 afile, chmod 4755 afile, chmod 2755 afile.

-if the file is executable, suid or sgid are represented by lowercase 's'. if it is not an executable, suid/sgid is uppercase 'S'.

-sticky bit - prevents 'world' users to delete files from a dir even if they have write permissions on the parent dir.

-sticky bit is represented as 't' or 'T' for world users. t = exec but no delete. T = neither exec nor delete (just like suid,sgid s or S).

-sticky bit is represented by octal 1.

-eg:           Kali$ chmod 1544 ab/bb/bbc

                 -r-xr--r-T 1 Kali staff 8 Apr 9 22:15 ab/bb/bbc

                 Kali$ chmod 1545 ab/bb/bbc

                 -r-xr--r-t 1 Kali staff 8 Apr 9 22:15 ab/bb/bbc

-file permissions (rwx) and access modes (sStT) apply only for non-root users.

-in other words, root users can delete files even if they dont have the permissions / access modes set so for the file.

-to prevent such accidents, the 'immutable flag' is used.

-immutable flag prevents even root from deleting files until the flag is unset.

-to set the immutable flag, chattr -+i cmd is used.

-to view the immutable flag, lsattr cmd is used.

-eg:         # chattr +i afile [ sets immutable flag for afile, even root cant delete it ]

               # lsattr afile [ shows immutable flag permi ]

               # chattr -i afile [ unsets immutable flag ]

-man capabilities for more on immutable flag.

-umask - permissions that a user does not want to grant automatically to newly created files / dirs.

-umask is like the octal-negative of file permissions. eg: umask 022 => default permi will be 755.

Linux HowTo: Redirection Basics

Linux (like Unix) comes with the very powerful command line option of input, output and error redirection. In simple terms it means taking the input and showing the output and/or error on non default sources. Note that the standard devices are Std in = 0, Std out = 1 and Std err = 2.

-To redirect both standard output and standard error into a file the following operator is used:

- &> or &>>

-eg      # cmd > output.txt 2>&1

-eg      # cmd > /dev/null 2>&1

-note: . this discussion didn't talk about redirecting standard input (only standard output and standard error).

. so this form of redirection works only for standard output and standard error.

-input redirection works like this:

           # cmd < afile

-eg:     # sort  < afile            [ it is just like sort abc ]

-'here document' is a special type of input redirection where a redirector is << and a word is used a marker for start and end.

          # cmd << keyword

              bla bla

           keyword

-eg:    # sort <<st                [ here the chosen keyword = st ]

          > a

          > Q

          > A

          > b

          > st

          A                                 [ <-- sorted output started from here ]

          Q

          a

          b

numbers

-pipelines ( a series of pipes ) only only work between stdin and stdout.

-stderr can not be piped.

 

eg:      # cat afile | grep 'test'  |  sort  |  uniq -c                       - sorts and counts unique occurances of word 'test' in file called afile.

Linux HowTo: MBR and Grub

The Grub boot loader can be installed in the

          . MBR or
          . the partition boot record of a partition or
          . on removable media (floppy, cd, usb key)

-grub config file is /boot/grub/grub.conf
-grub cmd /sbin/grub or /usr/sbin/grub is a small but powerful shell that supports several grub cmds.
-grub.conf is generated by anaconda, the linux installer.

-In the grub.conf file :

            . all counting in grub.conf starts with 0. eg 'default=2' implies 'default os = 3rd stanza'
            . splashimage = the background image for the grub boot menu.
            . root = partition that will be booted (ie /boot partition).
eg:   root (hd0, 6)     => /dev/hda7 = /boot partition.
       root (hd1, 10)   => /dev/hdb11 = /boot partition.
       root (hd2, 7)     => /dev/hdc8 = /boot partition.

- initrd => initial RAM disk => the disk partition that contains modules needed by kernel before file systems can be mounted.

-To install grub to a removable disk use the 'grub-install' cmd
-eg: for floppy disk:

            # grub-install /dev/fd0

-note: this loads the stage 1 boot loader (ie mbr) to the first sector of the floppy disk which loads stage2 boot loader ( ie grub, which lives on the hard disk)

-stage1 bootloader on floppy will still show empty when mounted as the first sector does not show up in the filesystem.
-stage1 bootloader only has a list of block addresses for stage2 bootloader.
-So if a partition address changes (say because of installation of a 2nd os on the system eg in a dual boot system), grub needs to be reconfigured in order for stage1 to locate stage2 bootloader.

-anyone having access to the grub cmd line also has access to files on the filesystems without the restrictions of file / owner permissions.
-the habit of creating a boot floppy or usb disk is good because it can help in case the mbr gets overwritten by another os install.
-even if the boot floppy or usb disk are not available, then linux install disk can be used to go in recovery mode and then mbr reinstalled.
-eg:      # chroot /mnt/sysimage [ on the recovery window, to make /mnt/sysimage as root mount directory ]

            # grub-install [ reinstalls mbr ]

-remember:
[[ 1st 512 bytes = mbr + partition table + bootloader (optional) ]]
[[ mbr = 1st sector = stage1 boot loader ]]
[[ bootloader = stage2 boot loader ]]

Linux HowTo: Disk Types and commands

Three prominent Disk types in Linux:

         . ide   - low cost, smaller     - stands for 'Integrated Drive Electronics', usually seen in home computers
         . pata - same as ide             - stands for 'Parallel Advance Tech Attachment (from PC - AT Days)

         . scsi  - bigger, better perf   - stands for 'Small Computer System Interface', usually seen in servers

         . sata  - improved ide          - stands for 'Serial ATA'

-To see disk info:

         # cat /dev/hdx - ide

    or  # cat /dev/sdx - scsi
         . /dev is a virtual fs like /proc & /sys.

-Note:
         . Although scsi / sata are 'serial' bus technologies as opposed to ide / pata which are 'parallel' bus tech, the performance of scsi / sata is generally better due to their superior engineering and bus size; scsi / sata is also more expensive than ide / pata.
        

-scsi :

          . was designed for connecting streaming devices like tapes and block devices like disks, cds, dvds etc. but now extended to printers/scanners.

          . was designed to allow multiple devices on one bus--the bus-controller controlling the data traffic.

-scsi defined by ANSI.

-useful files and cmds:

          # cat /proc/scsi/scsi, scsi_info, sginfo.

-USB

-usb 2.0 supports speeds upto 480 Mbps.

-usb cable has 4 wires - two for signals and one each for power and ground.

-To see usb info:

          # cat /proc/bus/usb

          # cat /proc/bus/usb/devices

-Linux filesystem is a single fs with the / as the top root dir.

-To view the different devices like floppy, cdrom etc, on the same fs, you mount them on their mount points.

-In this regard, different mount points are treated as different devices.

-This is different from Windows where every drive letter (A, B, C..) has its own fs.

-some example dirs under / :

          . bin     - essential cmd binaries

          . sbin    - essential system binaries

          . lib       - essential libraries & kernel modules.

-disk partition names:

           . /dev/hdx - ide

           . /dev/sdx - scsi / sata

-sector     = 512 bytes

-track      = sum (sectors) in one read of disk arm

-cylinder  = sum (tracks) in one read of disk arm

-Partition types:

           . primary - one of the 4 partitions limited by the master boot record (mbr); mbr resides in the 1st sector of the disk (ie the first 512 bytes).

           . extended - one of the primary partitions that is logically broken to create more than 4 partitions.

           . logical - one of constituents of the extended partition.

-The boot partition must be a primary partition and reside completely in the first 1024 cylinders;

-this is because the bios can't read or boot from the boot partition, if this condition is not met.

-usually 100 MB for boot partition is ok.

-partition recommendations:

            . first define boot

            . then define swap

            . then define root ( / ), /usr, /opt, /var in a single large partition (usually / )

            . after that define rest of the system like /home etc.

-Note: even if a system can have 4 primary partitions, it can still have more than 4 bootable os partitions; this is possible bec of boot-loaders.

-eg. of boot loaders = grub, lilo, bootmagic.

Linux HowTo: IOPorts, Interrupts, DMA

The computer, in order to communicate with the hardware devices connected to it, assignes and uses certain mechanisms like ports, interrupts, dma. These signals are transmitted over the buses on the motherboard..

-bus:

           . communication channels over which peripherals communicate with the cpu.

           . bus exist on the mother board.

           . current bus architecture is called PCI. PCI replaced ISA that came with PC-AT in 1984. Hence ISA was also called AT Bus.

           . PCI supports 32 bit devices. ISA supported upto 16 bit devices.

-To see PCI devices:

          # cat /proc/pci

      or # lspci

-ports:

           . When cpu needs to communicate to peripheral devices, it does so by writing to IO ports or simply ports.

           . each device has its own separate port that is not shared with other devices.

-To see the IO ports on the system:

          # cat /proc/ioports

-Note: ports are denoted in hexadecimal eg. 37A

-interrupts:

           . interrupts or IRQs are signals generated by peripherals for the cpu to know.

           . when the cpu receives an interrupt signal, it temporarily suspends all its activities and attends to the interrupt (thats why called 'interrupt').

           . interrupts may be shared between peripheral devices (unlike ports)

-To see interrupts:

           # cat /proc/interrupts - current interrupts

           # dmesg | grep -i share - to see shared interrupts besides other things.

-dma:

           . direct memory access

           . for faster peripheral devices, generating interrupts to talk to cpu and then gain access to RAM could slow things down.

           . dma solves this problem by granting the peripheral direct access to memory areas; basically bypassing the cpu.

           # cat /proc/dma

           # dmesg | grep -i dma - to see dma info besides other things.

-pnp

           . with the advent of newer and multitude of peripherals, ports and interrupts became scarce. for instance two devices could not share ports or irqs.

           . thats when plug n play (pnp) concept came into being.

           . pnp devices report a set of interrupts/ports they need for working at boot time to bios and the system then doles those ports/irqs on need basis.

           . These days, pnp is a standard feature of most computers.

Linux HowTo: Package Repositories and Commands

Contemporary Linux is based on two main camps of Software Package Types. One is based on the Red Hat Packages and the other on Debian. The popular Linux offering from Canonical called Ubuntu is based on the Debian packages. It was originally derived in 2004 from one of the Debian unstable releases and named 'Warty Warthog'. A new release is made available every 6 months; the latest one being 'Maverick Meerkat'. Here we discuss some commands to manage the packages in Linux..

===

-config file for client update repositories in ubuntu:
        . /etc/apt/sources.list
-pkg install dir in ubuntu:
        . /var/cache/apt/archives
-dpkg -l  =  rpm -qa = lists installed packages
-dpkg does not resolve dependencies.
-apt-get resolves dependencies.
-to update local package database (in /var/cache/apt/archives?):
        # apt-get update
-to compare against available upgrades:
        # apt-get upgrade
-to review available repositories:
        # apt-cache depends <pkgname>   
-note: apt-cache depends on the freshness of repositories, which is done by 'apt-get update'
-to review available repositories from install cdrom
        # apt-cdrom
-repository types:
        . main        - opensource, supported by cano
        . restricted    - proprietary, supported
        . universe    - opensource, not supp by cano, supp by wider ubuntu opensrc cmomm
        . multiverse    - proprietary, not supp by cano.
-local directories (or even ftp, http, nfs dirs) can be created as repositories.

-local repositories are configured in /etc/apt/sources.list
-it has cmds in pairs like this:
        . deb http://...        -specifies location for binaries
        . deb-src http://...        -specifies location for source code
-repository types are main, restricted, universe, multiverse.
-software downloads from mirror sites is possible in http, ftp or rsync.
-to open software download dialog:
        # software-properties-gtk
-alternate graphical methods:about:home
        # synaptic
or     # update-manager
-synaptic reload button = apt-get update equivalent
-above cmd updates the rep db /var/cache/apt/archives based on repos in /etc/apt/sources.list   

-sometimes it is advisable to setup a local network mirror which serves hosts in a lan.
-such a local mirror can be updated infrequently to a remote mirror using:
        . apt-mirror    to
 or    . rsync
-to create local mirror:
        # apt-get install apt-mirror       
        . configure /etc/apt/mirror.list & set the var base_path which points to local rep.

Linux HowTo: PAT, NAT, VPN

Before we jump into PAT, NAT and VPN, lets take a step back and begin with the technology behind many of these services called Port Forwarding...

-Port Forwarding-
    -Also called Port Mapping is the process of changing of the destination address and/or port on a packet.
    -Port forwarding on the router permits communications by external hosts with services provided within a private lan
    -this permits public hosts (eg on the Internet) to connect to a specific host within a private lan.
-scenarios of Port Forwarding:
    -running a public http server within a private lan at port 80
    -permitting ssh access to hosts on the private lan from the Internet at port 22
    -permitting ftp access to hosts on the private lan from the Internet at port 21.
-Port Forwarding is achieved by-
    -iptables cmd     in linux
   
-Note: 
    -Two cases of Port Forwarding are PAT (Port Address Translation) and NAT (Network Address Translation). PAT is subset of NAT.
    -NAT Translates IP addr only. 1to1 IP translation also called Static NAT.
    -PAT Translates IP addr + port (ie socket). Also called NAT overload.


    -In a typical home lan via a router, the Internet sees only the router which holds the public ip addr.
    -the hosts behind the router are invisible to the Internet.

    -PAT permits communication between hosts on a private n/w and hosts on a public n/w.
    -It allows a single IP addr to be used by many hosts on a private n/w.   
    -PAT device (usually router) transparently modifies IP packets as they pass through it.
    -PAT device modifies the senders IP Addr and Port number (to a public ip and port)
    -PAT is also known as NAT overload.
    -PAT operates on layer 3 & 4 (network, transport resp). NAT operates only on layer 3.

    -Socket    -ip + port pair        (much like a telephone line and its extn).
    -the socket needs to be known by both source and target host for communication to happen.


-VPN 
    -a n/w that uses a public telecom n/w like the Internet to provide remote network access.
    -the goal of vpn is to provide same level of security as a private n/w at a fraction of the cost.
    -vpns came in vogue in around Y2K when leased lines were the only option available and that too at a high cost.
    -vpns actually spelled the end of leased lines.
    -vpns provide security by encapsulating the traffic between the two nodes in cryptographic tunnels.
    -vpns use several protocols for providing security - eg ssh, ipsec (ip security), ssl etc.

-Tunneling protocol-
    -a n/w protocol that encapsulates payload of another n/w protocol.
    -this is routinely used in vpn.
    -tunneling usually has two protocols operating - the 'delivery protocol' that encapsulates the 'payload protocol'
-eg:    -delivery protocol = ssh, payload protocol = smb; ssh + smb = ssh tunneling protocol.




Note: In contrast to IP based Computer networks, traditional networks like Cable TV provides TV Broadcast in the form of Radio Freq Signals over optical fiber or coaxial cables.
    -This is different from traditional TV Broadcast via radio waves over-the-air.
    -Cable TV networks have a high bandwidth.

Linux HowTo: HAL Basics

Ever wondered how does your computer magically sense and install your usb drive or cd? Welcome to the world of 'plug and play' using the concept of HAL--Hardware Abstraction Layer--A software that sits between your kernel and hardware.
  
    -hal    - hardware abstraction layer.
    -automatic detection and initialization of hardware (plug n play) happens because of hal.
    -hal works under the daemon called 'hald'.
    -hald maintains a database of currently connected hardware.
    -hal      = sysfs    +  udev    +  hald    +  dbus
    -sysfs    = stores settings about hald
    -udev    = listens for new devices and kicks off procs; acts as the manager of files in /dev dir.
    -hald     = the daemon, maintains a database of h/w conn currently
    -dbus    = handshakes hardware and actual running proc.

Modules - Modules are snippets of code pertaining to a specific hardware. Modules are the way in which kernel manages a hardware. Modules can be loaded and unloaded on demand; this is most commonly seen in case of plug and play devices.

Useful Module commands:
    -lsmod         - lists currently loaded kernel modules.
    -lspci           - lists h/w connected via the pci bus.
    -lspcmcia & lsusb cmds behave similarly.
    -lshal and lshw show info about currently conn devices.

    -modprobe  - probes  modules
    -insmod       - inserts modules
    -rmmod       - removes modules

Linux HowTo: MBR, Grub, Lilo etc

Here is a discussion about Linux's MBR (Master Boot Record, Grub (Grand Unified Bootlader) and Lilo (Linux Loader)..

===

Partition types:

      . primary - one of the 4 partitions limited by the master boot record (mbr); mbr resides in the 1st sector of the disk (ie first 512 bytes).

      . extended - one of the primary partitions that is logically broken to create more than 4 partitions.

      . logical - one of constituents of the extended partition.

-the boot partition must be a primary partition and reside completely in the first 1024 cylinders;

-this is because the bios can't read or boot from the boot partition, if this condition is not met.

-usually 100 MB for boot partition is ok.

-partition recommendations:

      . first define boot

      . then define swap

      . then define /usr, /opt, /var in a single large partition - perhaps / ?

      . after that define rest of the system like /home etc.

-mbr lives in the first sector of the first primary partition. the mbr contains the partition table, info about the partitions in the system.

-Since a sector = 512 bytes, mbr = 512 bytes & in turn partition table = 512 bytes.

-every media (disk, floppy, cd) contains an executable code in the mbr even if the code is only to put a message "Non-bootable disk in drive A:".

-this is the code that is loaded by bios during the bootstrap. this is called 'stage1 boot loader'.

-this code from mbr / stage1 boot loader (ie first sector) looks for active primary partition and loads the first few blocks of that partition into ram.

-these few blocks from active primary partition comprise 'stage 2 boot loader'.

-stage 1 + stage 2 = boot strapping.

-the above works fine if there is only one os in the system. but if there are multiple os, then another piece of code called boot-loader is needed.

-the boot-loader allows the user to select one of the os to boot, ie choose which set of first os-disk-blocks to load into ram.

-Note: even if a system can have 4 primary partitions, it can still have more than 4 bootable os partitions; this is possible bec of boot-loaders.

-eg of boot loaders = grub, lilo, bootmagic.

-bootloader lives in an os partition and is invoked by the mbr. [[ (mbr.exe) ] --> (bootloader.exe) --> (rest of os partition) ]

-Why is grub > lilo?

Because when changes are made to the system (new os, new kernel) lilo boot-setup needs to be recreated from the cmd line whereas for

grub only the grub.conf file needs to be re-edited.

-Lilo can be installed in the

       . MBR or

       . the partition boot record of a partition or

       . on removable media (floppy, cd, usb key)

-lilo config file is /etc/lilo.conf

-Grub can be installed in the

       . MBR or

       . the partition boot record of a partition or

       . on removable media (floppy, cd, usb key)

-grub config file is /boot/grub/grub.conf

-grub cmd /sbin/grub or /usr/sbin/grub is a small but powerful shell that supports several grub cmds.

-grub.conf is generated by anaconda, the linux installer.

-In the grub.conf file :

       . all counting in grub.conf starts with 0. eg default=2 => 3rd stanza.

       . splashimage = the background image for the grub boot menu.

       . root = partition that will be booted (ie /boot partition).

eg:    root (hd0, 6) => /dev/hda7 = /boot partition.

         root (hd1, 10)=> /dev/hdb11 = /boot partition.

         root (hd2, 7) => /dev/hdc8 = /boot partition.

       . initrd => initial RAM disk => the disk partition that contains modules needed by kernel before file systems can be mounted.

-To install grub to a removable disk use the 'grub-install' cmd

-eg: for floppy disk:

       # grub-install /dev/fd0

-note: this loads the stage 1 boot loader to the first sector of the floppy disk which loads stage2 boot loader (which lives on the hard disk)

-stage1 bootloader on floppy will still show empty when mounted as the first sector does not show up in the filesystem.

-stage1 bootloader only has a list of block addresses for stage2 bootloader.

-So if a partition address changes, grub needs to be reconfigured in order for stage1 to locate stage2 bootloader.

-Anyone having access to the grub cmd line also has access to files on the filesystems without the restrictions of file / owner permissions.

-the habit of creating a boot floppy or usb disk is good because it can help in case the mbr gets overwritten by another os install.

-even if the boot floppy or usb disk are not available, then linux install disk can be used to go in recovery mode and then mbr reinstalled.

-eg: # chroot /mnt/sysimage [ on the recovery window, to make /mnt/sysimage as root mount directory ]

       # grub-install [ reinstalls mbr ]

-Remember:

[[ mbr = 1st sector = stage1 boot loader ]]

|-> stage2 bootloader partition 1 -> grub menu option 1

|-> stage2 bootloader partition 2 -> grub menu option 2

|-> stage2 bootl<<<oader partition 3 -> grub menu option 3

. . . . . .

Linux HowTo: Hardware Basics - 1

Some notes on Hardware basics on a Linux computer; more to follow soon..

===

Booting or bootstrapping:

           . The reprogramming of hardware to do tasks by turning power on.

           . This happens when BIOS loads itself into RAM, performs POST & calls the OS kernel and drivers into RAM.

           . BIOS resides in the form of ROM or EEPROM chips on the motherboard.

           . BIOS version and date can be seen in the BIOS menus. If it is very old, it is good to upgrade the BIOS. This can be done using a BIOS disk/flash drive etc.

Bus:

           . communication channels over which peripherals communicate with the cpu.

           . bus exist on the mother board.

           . current bus architecture is called PCI. PCI replaced ISA that came with PC-AT in 1984. Hence ISA was also called AT Bus.

           . PCI supports 32 bit devices. ISA supported upto 16 bit devices.

-To see PCI devices:

          # cat /proc/pci

      or # lspci

         Try and compare the above cmds.

-Ports:

           . When cpu needs to communicate to peripheral devices, it does so by writing to IO ports or simply ports.

           . each device has its own separate port that is not shared with other devices.

-To see the IO ports on the system:

          # cat /proc/ioports

-Note: ports are denoted in hexadecimal eg. 37A

-Interrupts:

           . interrupts or IRQs are signals generated by peripherals for the cpu to know.

           . when the cpu receives an interrupt signal, it temporarily suspends all its activities and attends to the interrupt (thats why called 'interrupt').

           . interrupts may be shared between peripheral devices (unlike ports)

-To see interrupts:

           # cat /proc/interrupts                         - current interrupts

           # dmesg | grep -i share                      - to see shared interrupts besides other things.

-Dma:

            . direct memory access

            . for faster peripheral devices, generating interrupts to talk to cpu and then gain access to RAM could slow things down.

            . dma solves this problem by granting the peripheral direct access to memory areas; basically bypassing the cpu.

           # cat /proc/dma

           # dmesg | grep -i dma - to see dma info besides other things.

-Plug n Play (pnp)

            . with the advent of newer and multitude of peripherals, ports and interrupts became scarce. for instance two devices could not share ports or irqs.

            . thats when plug n play (pnp) concept came into being.

            . pnp devices report a set of interrupts/ports they need for working at boot time to bios and the system then doles those ports/irqs on need basis.

            . These days, pnp is a standard feature of most computers.

-Disk types:

            . ide         - low cost, smaller

            . scsi        - bigger, better perf (usually in servers)

            . sata        - improvement over ide

-To see disk info:

           # cat /dev/hdx - ide

       or # cat /dev/sdx - scsi

-Note: /dev is a virtual fs like /proc.

-Scsi (said scuzzy) (Small Computer System Interface)

            . was designed for connecting streaming devices like tapes and block devices like disks, cds, dvds etc. but now extended to printers/scanners.

            . was designed to allow multiple devices on one bus--the bus-controller controlling the data traffic.

            . scsi standard defined by ANSI.

-useful:

            . cat /proc/scsi/scsi, scsi_info, sginfo.

-Usb 

            . Universal Serial Bus 

            . USB version 2.0 supports speeds upto 480 Mbps.

            . usb cable has 4 wires - two for signals and one each for power and ground.

-To see usb info:

           # cat /proc/bus/usb

           # cat /proc/bus/usb/devices

Extras:

-Linux filesystem is a single fs with the / as the top root dir.

-To view the different devices like floppy, cdrom etc, on the same fs, you mount them on their mount points.

-in this regard, different mount points are treated as different devices.

-This is different from Windows where every drive letter (A, B, C..) has its own fs.

 

Linux HowTo: Common Network Diagnostic Tools

Here is a short description of five common network diagnosing tools available in Linux:

       . ping           - simple icmp pkt to check n/w conn

       . traceroute  - sends a series of 3 test packets to each host en route between two hosts.

       . tracepath   - similar to traceroute but produces one line for each pkt so more verbose.

       . netstat       - useful for several places (see below)

       . tcpdump    - a packet sniffer, ie intercepts n/w pkts and log them or display them

-Examples

        # ping -c 10 ms_comp                          - c = count

        # traceroute -n ms_comp                      - n = numeric ip instead of hostnames - useful when dns has probs.

        # netstat -i ethn                                   - interface ethernet status

        # netstat -r                                           - routing table; combine with -n (eg netstat -rn) for numeric IP report

        # netstat -M                                         - show ip masquerading info.

        # netstat -a                                           - shows all ports, esp open ports

Note:

          -if local ping is successful but no response from destn => router prob. 

          -if hostname resolve not happening                              => dns prob.

          -if times are highly variable or missed times                => busy router or physical dist bet routers too great.

          -if lost pkt in first hop                                                  => local prob. if lost pkt in last hop => remote prob.

          -if ping works but traceroute doesnt                            => router is programed to drop traceroute pkts.

-tcpdump  - Dumps TCP/IP Traffic between hosts

           -tcpdump - low level packet sniffer. to be used with care after permissions from your manager as it can sniff sensitive data.

           -tcpdump is run as root.

           -tcpdump outputs one line for each packet it monitors. sometimes the lines are long and wrap around on the screen.

-eg:     # tcpdump [-c n]                            - count n packets

           # tcpdump -v                                 - verbose or -vv for very verbose

Note: 
- Like tcpdump, there is a graphical alternative called wireshark. Wireshark can read and graph tcpdump output.

- Packet Sniffers like tcpdump and wireshark can work even if they not running on sender / recvr. This means anyone having access to your network can sniff packets between any two nodes. So some organizations forbid running packet sniffers altogether--so take permission from your manager before you run tcpdump, wireshark, nmap etc.

Linux HowTo: Hardware clock, Software Clock, NTP

Two clocks exist in Linux computers:

1- hardware clock or bios clock or real time clock (RTC) 

      - maintains time while system is off

      - oscillating quartz crystal in the motherboard

      - it is accurate to a few seconds per day.

      - it is variable under conditions of temperature.

      - usually set to Greenwich Mean Time (GMT) or Universal Coordinated Time (UTC)

2- software clock or system clock 

      - set at startup, using hardware clock and then maintained on its own.

      - managed by the OS (in this case Linux)

      - it is variable under high system load and interrupt latency

      - a running linux system uses the software clock for most purposes.

Note: 

-in the networked world, a system that runs services across time zones, the linux system s/w clock is best set to UTC.

-but after that, they need to be aware of their time zone.

-eg: two networked systems, one in NY one in LA, have their software clock = UTC but timezone = East & Pacific reso,

-so they add / subtract appropriately to UTC when working.

-On a dual boot system also, hwclock is set as UTC to keep time mainly because the other OS (eg Windows) that are not UTC aware like Linux

-if s/w clock is UTC, set h/w to UTC also.

-The system reads the hardware clock at startup and from then on uses the software clock.

-the date cmd shows and sets the software clock not the hardware clock.

-the hwclock cmd shows the hardware clock.

       # hwclock --systohc                  -sets the h/w clock on the basis of the s/w clock

       # hwclock --hctosys                  -sets the s/w clock on the basis of the h/w clock

-note: . h/w clock can be set using BIOS also.

NTP or Network Time Protocol

         . network time protocol - a protocol to sync computer clocks over a network, usually over Internet, to UTC.

         . time sync is done by sending messages to time servers.

         . time returned is adjusted by a half the round trip delay, so the accuracy depends on latency in the network.

         . due to latency factor, the shorter the path to the time server, the more accurate the time keeping is.

         . due to the huge number of computers on the Internet, the load on the time servers can be huge.

         . to ease the load on the time-servers, the time servers are organized into strata, each stratum server serving about 1000 clients.

- stratum 0 - highly accurate atomic clocks

- stratum 1 - clocks directly depending on stratum 0

- stratum 2 - get their time from stratum 1 servers and make it available to stratum 3 servers.

- stratum 3 - get their time from stratum 2 servers and then make it available to next stratum and so on...

- even stratum 4 servers are accurate to subsecond times. accuracy goes down with each strata.

 

-NTP Configuration

          . ntp config info is maintained in /etc/ntp.conf

          . ntp is managed by ntpd daemon.

-useful ntp cmds:

         # system-config-time

         # ntpdate <server name>