Datastream

2023-09-27T14:05:01Z

Admin:

The Datastream Terminal is used for remote computing. Since only one University member can access it at a time, a time slot has to be booked before hand.

DATASTREAM has been retired - now using REFINITV's EIKON platform, which has now become the London Stock Exchange Group (LSEG).

Please read the simplified 2 step access: [https://vdc.univie.ac.at/ -> Refinitiv/Eikon :: access]

2021-09-17T12:04:21Z

Admin: /* Step by Step */

The High Performance Cluster of WiWi
== How To for users ==
See [[Slurm]].

== How To for Group Admins ==
Groups are independently administered by their group admins.
To indicate the group-spaces, the User-IDs (uid) are the Group IDs (gid) plus some addition.

I would suggest prefixing the users with ag_(+2 letters of group).

A new user in ag_hautsch should be ag_ha-USERXYZ. Her uid=7000+X, and her home-directory /home/ag_ha-USERXYZ.

===Specs===
Gruppe Prof. Krivobokova: Hr. Stefan Geissler (stefan.geissler@univie.ac.at)
Gruppe Prof. Cuchiero: Hr. Guido Gazzani (guido.gazzani@univie.ac.at)
Gruppe Prof. Ehmke: Hr. Rico Kötschau (rico.koetschau@univie.ac.at)
Gruppe Prof. Dörner: Roland Braune, Georg Fröhlich (georg.erwin.adrian.froehlich@univie.ac.at)
Gruppe Operations Research: Michael Kahr (m.kahr@univie.ac.at)
===implementation===
ag_testAG_ou ag_testAG (gid=9000) andreas.loibl@univie.ac.at uid=9000 + X

DN: uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
DN: all the other admins as well

ag_krivobokova_ou ag_krivobokova (gid=8000) stefan.geissler@univie.ac.at uid=8000 + X

DN: uid=ag_kr-geissler,ou=ag_krivobokova_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_cuchiero_ou ag_cuchiero (gid=3000) guido.gazzani@univie.ac.at uid=3000 + X

DN: uid=ag_cu-gazzani,ou=ag_cuchiero_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_ehmke_ou ag_ehmke (gid=4000) rico.koetschau@univie.ac.at uid=4000 + X

DN: uid=ag_em-koetschau,ou=ag_ehmke_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_doerner_ou ag_doerner (gid=5000) georg.erwin.adrian.froehlich@univie.ac.at uid=5000 + X

DN: uid=ag_do-froehlich,ou=ag_doerner_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

roland.braune@univie.ac.at
DN: uid=ag_do-braune,ou=ag_doerner_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_operres_ou ag_operres (gid=6000) m.kahr@univie.ac.at uid=6000 + X

DN: uid=ag_op-kahr,ou=ag_operres_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_hautsch_ou ag_hautsch (gid=7000) andreas.loibl@univie.ac.at uid=7000 + X

DN: uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
=== Step by Step ===
Group admin is within the university's network.

If you want to test a bit, '''ag_testAG_ou''' is the playground to do so.
* Group Admin can reach the admin interface: https://hpc3.wiwi.univie.ac.at
* Group Admin can log into the Web-Interface , e.g.
uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
password 12345
[[File:Login.png|none|300px|Login]]
If you might want to change your password, click your uid:
[[File:Change-password.png|none|300px|Change Password]]
* Connect via ssh:
ssh ag_ha-loibl@hpc3.wiwi.univie.ac.at
* Within this ssh session, Group admin can access OpenLdap directory entry (not the local /etc/passwd!)
getent passwd | grep ag-XY
[[File:Ssh-session.png|none|300px|SSH]]
If you assigned an uid of 9001, 9002, or 9003 to a new user, or /home/ag_te-testuser-1 as a home directory, or use a username of ag_te-testuser-1, then you would clearly violate the uniqueness, because all these attributes have already been used.
* go to blueprint user
[[File:Goto-testuser.png|none|300px|GTO Blueprint User]]
* Copy blueprint user to newuser
[[File:Copy-to-new-user.png|none|300px|copy to new user]]
* Fill in attrinutes for new user
[[File:New-entry-1.png|none|300px|attributes for new user]]
[[File:New-entry-2.png|none|300px|attributes for new user]]
[[File:New-entry-3.png|none|300px|attributes for new user]]

* Check on new user and submit
[[File:Check-and-submit.png|none|300px|check and submit]]
* OOps! Attribute (uid=9003 was not unique); Error message
[[File:Attribute-not-unique.png|none|300px|Error Message]]
* Make attribute unique (uid=9004)
[[File:Change_for_success.png|none|300px|Error Message]]
* Success!
[[File:Success2.png|none|300px|Success]]

* the researcher can now log in the web-interface and change her password.
* the researcher can now ssh to hpc3, the according home directory will be created.
* the researcher can transparently access the fat but a bit slower storage via
cd /group.storage/ag_YOUR-GROUP

2021-09-16T13:09:55Z

Admin: /* How To for Group Admins */

The High Performance Cluster of WiWi
== How To for users ==

== How To for Group Admins ==
Groups are independently administered by their group admins.
To indicate the group-spaces, the User-IDs (uid) are the Group IDs (gid) plus some addition.

===Specs===
Gruppe Prof. Krivobokova: Hr. Stefan Geissler (stefan.geissler@univie.ac.at)
Gruppe Prof. Cuchiero: Hr. Guido Gazzani (guido.gazzani@univie.ac.at)
Gruppe Prof. Ehmke: Hr. Rico Kötschau (rico.koetschau@univie.ac.at)
Gruppe Prof. Dörner: Roland Braune, Georg Fröhlich (georg.erwin.adrian.froehlich@univie.ac.at)
Gruppe Operations Research: Michael Kahr (m.kahr@univie.ac.at)
===implementation===
ag_testAG_ou ag_testAG (gid=9000) andreas.loibl@univie.ac.at uid=9000 + X

DN: uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
DN: all the other admins as well

ag_krivobokova_ou ag_krivobokova (gid=8000) stefan.geissler@univie.ac.at uid=8000 + X

DN: uid=ag_kr-geissler,ou=ag_krivobokova_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_cuchiero_ou ag_cuchiero (gid=3000) guido.gazzani@univie.ac.at uid=3000 + X

DN: uid=ag_cu-gazzani,ou=ag_cuchiero_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_ehmke_ou ag_ehmke (gid=4000) rico.koetschau@univie.ac.at uid=4000 + X

DN: uid=ag_em-koetschau,ou=ag_ehmke_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_doerner_ou ag_doerner (gid=5000) georg.erwin.adrian.froehlich@univie.ac.at uid=5000 + X

DN: uid=ag_do-froehlich,ou=ag_doerner_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

roland.braune@univie.ac.at
DN: uid=ag_do-braune,ou=ag_doerner_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_operres_ou ag_operres (gid=6000) m.kahr@univie.ac.at uid=6000 + X

DN: uid=ag_op-kahr,ou=ag_operres_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at

ag_hautsch_ou ag_hautsch (gid=7000) andreas.loibl@univie.ac.at uid=7000 + X

DN: uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
=== Step by Step ===
Group admin is within the university's network.
* Group Admin can reach the admin interface: https://hpc3.wiwi.univie.ac.at
* Group Admin can log into the Web-Interface , e.g.
uid=ag_ha-loibl,ou=ag_hautsch_ou,ou=People,dc=hpc,dc=wiwi,dc=univie,dc=ac,dc=at
password 12345

you might want to change your password.
* Group Admin can connect herself via ssh as well:
ssh ag_ha-loibl@hpc3.wiwi.univie.ac.at
* Within this ssh session Group admin can access OpenLdap directory entry (not the local /etc/passwd!)
getent passwd
here you can check if the uid, /home/xyz userdirectory, for the new user you want to create is really unique! (If you would assign some attribute breaking the uniqueness, the web-interface should throw an ERROR anyway.)

to be continued ...

HPC

2021-09-16T12:27:57Z

Admin: /* How To for Group Admins */

2021-07-14T13:31:17Z

Admin: /* Virtual Desktop */

CRSP

2021-07-14T13:06:33Z

Admin:

This page is for CUPL-Tools only. For the Windows desktop application -> [[CRSPSift]].
==CRSP, COMPUSTAT, CCM ==
=== CUPL Tools ===
==== access to your remote text terminal ====
===== require credentials =====
email the VDC sysadmin to provide credentials and get your remote text terminal prepared.
<ul>
<li>
host: crsp.vdc.univie.ac.at
</li>
<li>
user: your username
</li>
<li>
password: your password
</li>
</ul>

==== VPN ====
If working from home you have to establish a Virtual Private Network (VPN) to Uni first ([https://zid.univie.ac.at/en/vpn/ see ZID/VPN]).

==== SSH ====
If working from Windows, you might need to install an SSH-client ([https://www.putty.org/ like PUTTY]) to use Secure Shell (SSH) to connect to your terminal. Accept the server key.
[[File:3_accept_key.png|left|frameless|200px|caption]]
Also see [https://zid.univie.ac.at/online-speicherplatz/anleitungen/sftp/ ZID's explanation]
 

==== usage of a terminal ====
Now you work remotely and use the remote host's command line (which is the bash shell, don't type the beginning "$", which is the prompt):
[[File:4_terminal_session.gif|left|]]
 
the midnight-commander (mc) provides a built-in editor mcedit (exit it with Alt+0)
<pre>$ mc </pre>
<pre>$ mcedit ts_samp7.rqt </pre>
e.g. use '''''ts_print''''' to process a requirementfile '''''ts_samp7.rqt''''' and get the output '''''ts_samp7.out''''':
<pre>$ ts_print ts_samp7.rqt</pre>


use mc / mcedit to look into ts_samp7.out:
<pre>$ mcedit ts_samp7.out </pre>
[[File:8_ts_samp6_out.png||frameless|400px|caption]]
 
 
use e.g. this command to get the switches for the command '''''crsp_print''''':
<pre>$ crsp_print /d1 /var/crsp/2019-07/crspdata/cmz201807</pre>

==== download the results onto your desktop / laptop ====
use an SFTP client like [https://cyberduck.io/download/ CyberDuck] on a Mac or [https://zid.univie.ac.at/online-speicherplatz/anleitungen/sftp/ WinScp] on Windows to download the results (e.g. ts_samp7.out) onto your desktop for further processing. Provide your SSH credentials:

[[File:10_cyberduck_credentials.png|400px]]
 

==== documentation====
Please read some manuals to gain experience with the ts_sampleX.rqt files in your home directory
<ul>
<li>
[http://www.crsp.org/products/documentation/cupl-386-guide CRSP Utilities and Program Libraries (CUPL)]
</li>
<li>
[https://w3.loibl.com/uni/ccm_data_guide_0.pdf CCM Data Guide]
</li>
<li>
and maybe the [https://w3.loibl.com/uni/xf_understanding_the_data.pdf xf-understanding_the_data guide].
</li>
<li>
collected [https://w3.loibl.com/wiki/index.php/CRSP#Usage thoughts] .
</li>
</ul>
=== WEB Access R ===
* apply for login-credentials andreas.loibl@univie.ac.at
* use sftp to upload / download your date (like with your [https://zid.univie.ac.at/online-speicherplatz/anleitungen/sftp/ Online-Speicherplatz])
* http://calc.isor.univie.ac.at:8787
* explore some commands, for the sake of functionality only (not to make any sense)
###### Example#1
# search for "EGSA"and perform some actions
# comment, not interpreted
? system
? system.time()
help("crsp.data.usstock") ## we have licensed US-Stock 1925 and 4 US-Indices and CCM
system2("uptime") ## load average < 10 ! (otherwise too many concurrent users)
library(data.table, crsp)
setwd('~/crsp')
getwd()
myheaders <- data.table(crsp.data.usstock('sfz_hdr'))
myEGASHeaders <- myheaders[HTSYMBOL %like% "EGAS" & BEGDT > "1986-01-8"]$KYPERMNO
myMonthly <- crsp.data.usstock("sfz_mth")
nrow(myMonthly)
ncol(myMonthly)
head(myMonthly,10)
MyEGASData <- myMonthly[myMonthly$KYPERMNO %in% c(myEGASHeaders),]
head(MyEGASData)
tail(MyEGASData)
plot.ts(MyEGASData$MPRC)
######## Example #2
# read tickers from one file perform some operations and write results to another file
# myTechies <- read.csv('/home/su/crsp/techcompanies.tickers', header = FALSE)
myTechies <- scan(file = 'techcompanies.tickers', what = 'character', sep = ',')
myTechiesHeaders <- myheaders[myheaders$HTSYMBOL %in% c(myTechies)]
#fileDsk <- tempfile("MyTechiesHeaders")
fileDsk <- "MyTechiesHeaders"
#setwd("~/R_workspace/from_uspace/privat/crsp_sample1/")
write.table(t(myTechiesHeaders), file='./MyTechiesHeaders')
# file.show(fileDsk)
# if(interactive()) file.show(fileDsk)
# unlink(fileDsk) # delete file again
myTechiesData <- myMonthly[myMonthly$KYPERMNO %in% c(myTechiesHeaders$KYPERMNO),]
plot.ts(log(myTechiesData$MPRC))

==== Documentation ====
[http://www.crsp.org/products/documentation/overview-4 in the crsp.org universe.]

CRSPSift

2021-07-14T13:01:50Z

Admin:

The CrspSift Terminal is used for remote computing. A time slot has to be booked before hand.

=Overview=
== Install Clients ==
You might need to '''install 2 clients''' on your machine:
# A VPN client, an up to date (F5 BigIp) VPN-Client ([https://zid.univie.ac.at/en/services/services-from-a-z/v/vpn-virtual-private-network/ ZID])
# A VNC client, to connect to the desktop at the datacenter, like tigervnc (former tight-vnc),
[[File:linux_icon.png|14px]] linux: yum install tigervnc ''or'' apt-get install tigervnc-viewer,

or similar viewers for windows, etc.

== Steps ==
There are a few '''steps to follow''':
# Stay within the UNI's network (if working from hone, use the VPN).
# Book a time slot for your session
# Once your time slot has started, retrieve your password
# Connect to the data-center's desktop using a tiger/tight vnc-viewer
= Steps =
==VPN==
Once connected via VPN, you could check if you are up by reaching an internal server like [http://calc.isor.univie.ac.at:8787 Calc]

==Book Session==
Book [https://shib.isor.univie.ac.at/booking/CRSPSift/calendar.php a time slot] "Fair-play" is assumed, restrict yourself to a couple of hours per week not to get restricted.

==Retrieve Password==
After your time slot has begun, [https://shib.isor.univie.ac.at/booking/CRSPSift/show_password.php retrieve your password.]

== Virtual Desktop==
[[File:VirtualDesktop.gif|right|frameless|100px|caption]]
# After you have already connected to the UNI's VPN,
[[File:Vncviewer_crspsift.jpg]] click '''<Connect>''' [https://shib.isor.univie.ac.at/booking/CRSPSift/show_password.php and provide password.] [[File:Vnc_passwd.png]]

[[File:linux_icon.png|14px]]: run
vncviewer crspsift.vdc.univie.ac.at:0
and authenticate with your [https://shib.isor.univie.ac.at/booking/CRSPSift/show_password.php session-password.]

Apple might require to fill in '''crspsift.vdc.univie.ac.at:5900'''



Log on as '''Researcher'''. A new Desktop will be prepared for you, which could take up to two minutes.

=Applications=
==Excel==
You can query and retrieve your results directly from within Excel through plugins by using the icon at the upper left corner of your desktop.

==Storage Clients==
Various clients have been installed for saving your data, like Google-drive, u:cloud or Dropbox. Don't forget to save your work before your session ends. Everything will be deleted, so that the next user starts without any left-overs.
=Documentation =
* [http://www.crsp.org/files/Sift_guide.pdf User Guide]

CRSPSift

2021-07-14T12:54:10Z

Admin:

The CrspSift Terminal is used for remote computing. A time slot has to be booked before hand.

=Overview=
== Install Clients ==
You might need to '''install 2 clients''' on your machine:
# A VPN client, an up to date (F5 BigIp) VPN-Client ([https://zid.univie.ac.at/en/services/services-from-a-z/v/vpn-virtual-private-network/ ZID])
# A VNC client, to connect to the desktop at the datacenter, like tigervnc (former tight-vnc),
[[File:linux_icon.png|14px]] linux: yum install tigervnc ''or'' apt-get install tigervnc-viewer,

or similar viewers for windows, etc.

== Steps ==
There are a few '''steps to follow''':
# Stay within the UNI's network (if working from hone, use the VPN).
# Book a time slot for your session
# Once your time slot has started, retrieve your password
# Connect to the data-center's desktop using a tiger/tight vnc-viewer
= Steps =
==VPN==
Once connected via VPN, you could check if you are up by reaching an internal server like [http://calc.isor.univie.ac.at:8787 Calc]

==Book Session==
Book [https://shib.isor.univie.ac.at/booking-crspsift/calendar.php a time-slot.] "Fair-play" is assumed, restrict yourself to a couple of hours per week not to get restricted.

==Retrieve Password==
After your time slot has begun, [https://shib.isor.univie.ac.at/booking-crspsift/show_password.php retrieve your password.]

== Virtual Desktop==
[[File:VirtualDesktop.gif|right|frameless|100px|caption]]
# After you have already connected to the UNI's VPN,
[[File:Vncviewer_crspsift.jpg]] click '''<Connect>''' [https://shib.isor.univie.ac.at/booking/CRSPSift/show_password.php and provide password.] [[File:Vnc_passwd.png]]

[[File:linux_icon.png|14px]]: run
vncviewer crspsift.vdc.univie.ac.at:0
and authenticate with your [https://shib.isor.univie.ac.at/booking/CRSPSift/show_password.php session-password.]

Apple might require to fill in '''crspsift.vdc.univie.ac.at:5900'''



Log on as '''Researcher'''. A new Desktop will be prepared for you, which could take up to two minutes.

=Applications=
==Excel==
You can query and retrieve your results directly from within Excel through plugins by using the icon at the upper left corner of your desktop.

==Storage Clients==
Various clients have been installed for saving your data, like Google-drive, u:cloud or Dropbox. Don't forget to save your work before your session ends. Everything will be deleted, so that the next user starts without any left-overs.
=Documentation =
* [http://www.crsp.org/files/Sift_guide.pdf User Guide]

Datastream

2021-07-13T13:52:44Z

Admin: /* Access / Workflow */

The Datastream Terminal is used for remote computing. Since only one University member can access it at a time, a time slot has to be booked before hand.
=Functional Range=

Datastream gives you access to the world's largest and most respected historical financial numerical database such as
# Worldwide equity coverage direct from the stock markets
# Comprehensive market indices,
# Economics data direct from national government sources as well as the OECD and IMF,
# Fixed income securities and associated indices, commodities and derivatives data.
# Forecast earnings data from IBES, fundamental data from Worldscope, and
# Added value data sets from respected sources such as MSCI, DJ Stoxx and FTSE All World.

=Overview=
== Install Clients ==
You might need to '''install 2 clients''' on your machine:
# A VPN client, an up to date (F5 BigIp) VPN-Client ([https://zid.univie.ac.at/en/services/services-from-a-z/v/vpn-virtual-private-network/ ZID])
# A VNC client, to connect to the desktop at the datacenter, like tigervnc (former tight-vnc),
[[File:linux_icon.png|14px]] linux: yum install tigervnc ''or'' apt-get install tigervnc-viewer,

or similar viewers for windows, etc.

== Steps ==
There are a few '''steps to follow''':
# Stay within the UNI's network (if working from hone, use the VPN).
# Book a time slot for your session
# Once your time slot has started, retrieve your password
# Connect to the data-center's desktop using a tiger/tight vnc-viewer
= Steps =
==VPN==
Once connected via VPN, you could check the functionality by just connecting to [http://calc.isor.univie.ac.at:8787 Calc]

==Book Session==
Because our license is "per seat", [https://shib.isor.univie.ac.at/booking-datastream/calendar.php you have to book a time-slot.] "Fair-play" is assumed, restrict yourself to a couple of hours per week not to get restricted.

==Retrieve Password==
After your time slot has begun, [https://shib.isor.univie.ac.at/booking-datastream/show_password.php retrieve your password.]

== Virtual Desktop==
[[File:VirtualDesktop.gif|right|frameless|100px|caption]]
# After you have already connected to the UNI's VPN,
# start your vnc-viewer and use your [https://shib.isor.univie.ac.at/booking/datastream/show_password.php session-password] to establish a connection
[[File:vncviewer.png]] click '''<Connect>''' [https://shib.isor.univie.ac.at/booking/datastream/show_password.php and provide password.] [[File:Vnc_passwd.png]]

Apple might require to fill in '''datastream.isor.univie.ac.at:5900'''

[[File:linux_icon.png|14px]] vncviewer datastream.isor.univie.ac.at:0



Log on as '''Researcher'''. A new Desktop will be prepared for you, which could take up to two minutes.

=Applications=
==Excel==
You can query and retrieve your results directly from within Excel through plugins by using the icon at the upper left corner of your desktop.

==Storage Clients==
Various clients have been installed for saving your data, like Google-drive, u:cloud or Dropbox. Don't forget to save your work before your session ends. Everything will be deleted, so that the next user starts without any left-overs.
=Documentation =
* [https://w3.loibl.com/uni/datastream_5_1_getting_started.pdf getting started]
* [https://w3.loibl.com/uni/datastream_guide_excel2010.pdf use Excel]