Augmented Reality Project

2006-06-22T16:52:00.000-07:00

Things have been going good over the past few weeks I have got alot done. My total hours so far this summer has been 60 hours. These past few weeks I have been trying to get the fickr onto the blogg, I still can not figure out how to do that but I am going to try a few more things. I have read the paper and learned alot of information. I new the the basics of alot of it but it was good to read it all in depth I have printed them off and highlighted things that I did not know and the things that I found interesting. Because I have not taken the course on local anesthia alot of that information was new to me. I did not know that some dental school have instigated the use of stimulated patients called DentSims to serve as models for skills application. The DentSims mannequins of the head and neck that have the same intra-oral anatomical features of an acual human patient. But the problem is they cant see intra-oral clinical features. A huge benifit to the 3D virtual objects will be that the studnet would be able to see their technique for anesthia from needle penetration to the deposition of the anesthetic at the correct anatomical site. This would allow them to experience visual and sensory percepts and develop relationships between theory and practice, thus minimizing the trail and error process for successful administration of local anesthia. The purpose of artifical environments is to over come these difficulties while at the same time maintaining the ability to teach abstractions directly through reification. In addition, as the students will experience haptic feedback related to the penetration of soft tissue, osseous contact, aspiration and depostion of anesthetic. There is alot more I have learned and have it all highlighted and summerized on the papers. I am going to continue working on the flickr and continue searching through the diff. web sites this week.

2006-05-31T09:10:00.000-07:00

Things have been going well. I have been learning alot. I have created a Flickr with the pictures from Orlando, but I am still trying to firgure out how to put the Flickr that I have created onto the blog. So this week and the end of next week I hope to have that loaded onto the blog. I have also cont. to look at the different web sites and have learned alot. I am still reading the papers that Proff. Hanson has written, my goal is to have them both read and summerized by what I already knew and what I have learned and post it next week. I have spent 6 hours this week so far.

2006-05-23T10:36:00.000-07:00

At the Dental Educators Meeting the project was accepted to go to Orlando! It was also recoginzed at the Donor Lunch In for undergraduate research.

2006-05-23T09:51:00.000-07:00

The Senior have graduated and are on their way out into the working field of Dental Hygiene. They have researched and done alot for the Augmented Reality project. I and two other student are going to continue their findings on this project over the next year.

I have just started with this project and I am going to continue the research over the summer. My objectives to begin with are as follows:
1) To get familiar with the Mac- I have been playing around with it, it is alot different then what I am used to. It's hard to get the hang of it but hopefully with time it will get better.
2) I am also going to explore the different web sites we are using for this project, to help me get familiar with those.
3) I have started to read two papers that Prof. Hanson has written about the project so that I have a better understanding and knowledge about the project.
4) I am going to figure out how to add pictures to the blogg and create a flicker of the pictures from Orlando and post them on the blogg, and also change the template of the blogg.

Our Future Objectives are to:
1) Look at Jazz Studio and make our own 3D objects
2) Investigate visual and magnetic trackers
3)Learn about plug ins

I have alot to get working on as you can see. I am excited and a little overwhelmed with it because I dont have alot of knowledge to start out with and I have alot to figure out but I excited to read about it and continue the research. I have spent about 6 hours so far helping at the lunch in and doing some reading on my own time.

2005-11-14T14:24:00.000-08:00

We have returned from the summer and are now in the fall semester 2005. We submitted a proposal for funding to purchase equipment for our project. We will be purchasing the "flock of birds" sensor system and the one of the omni haptics packages. We will also purchase our 3D image from Zygote. We also made a proposal for funding to send Marie, Heidi and I to the ADEA conference in Orlando, Florida. We have not yet heard back from the conference or the undergraduate research board. We hope to hear confirmation around the first week of December.

We have come across a step in our project that we overlooked. Our AR team and the independent computer science consultants and Brett Shelton from Utah State all thought that when we purchased the Flock of Birds sensor package, we would also receive the appropriate software, "rendering engine," that would basically integrate all the different parts. We were mistaken and we will not get the software from the Flock of Birds. This poses a problem for the continuation of our research.

Rendering is defined as taking a set of data points (XYZ) in a virtual world nd drawing them on the computer screen in a 2D image. Our project entails taking a 2D image seen through the HMD and trying to make it into a 3D image. It is relatively easy to program around three data points from one source but it is more difficult to take data points from different sources and incorporate them into one image. This means that we need some institute that is either using a rendering engine all ready programed that we can use at our University, or incorporate a computer science students from a University that has a rendering engine to come on board our project, it also means that we could hire a programmer that would be full time on our project. These are just ideas, we have not made any decisions at this time but we will need to in the next couple of weeks.

An important part of our haptics and sensor system is having the computer able to recognize a change in material on our 3D object and then having the information from the computer able to be recognized by the stylus/ haptics piece. When choosing sensors, we need to know if the sensors are able to give a set of data points to say "what is where." We also need to find our how good our software is, we now know that we have no software.

The combination of the two should result in "relative position." meaning, Our image and the computer tells us in"real time" where the stylus is in position to the head.

According to one of our Independent consultants, Lynden Jones, if we do not have the appropriate software, which we do not, it would take longer then our January deadline to program software to our specific requirements. However, we have come across a "rendering engine" that we can purchase from sense8.com called worldtoolkit, costing around $10,000. We have another opportunity to request funding, deadline Dec 1, 2005, if that is what needs to happen for our project to continue.

2005-08-09T14:23:00.000-07:00

Today is our last day of the summer extetion of our research project. We will continue our research starting fall semester 2005. We will also hopefully implementing our project during the local anesthesia courses starting in October 2005. Today we discussed how we thought our research was progressing and any ideas of how we could improve during the fall. We will be including in our research team Heidi Law, another Dental Hygiene undergraduate research student who will be taking over the majority of the project during the fall.

We have submitted proposals for further funding from our specific college at Weber State University and we will continue to wait for any kind of word from them. We will then be able to purchase the equipment needed to implement.

The undergraduate research students on the team have started to write their bachelor thesis paper and should be able to complete the entirity of it except for the results section which will be in October 2005.

We have submitted to numerous research conferences as mentioned in past blogs and hope to be accepted to present our research at one or all of them.

Our knowledge of augmented reality/ mixed reality has significantly expanded and our research project is well underway for implementation. We feel that our summer hours have been spent very productively and that our project can benefit the dental hygiene department and can be expanded eventually to benefit other healthcare departments at Weber State University.

This blog will continue when the fall semester 2005 begins.

2005-08-04T14:12:00.000-07:00

Today we went to Utah State University and met with Asst. Prof. Brett Shelton, PhD. He was very patient and helpful in sharing his views on where we stand with our project and where we should be headed in the future. We really appreciate his willingness to share his time and his vast knowledge with us! Here are a number of things we discussed with Prof. Shelton.

There are different advantages and disadvantages of DAZ studio vs. Zygote for creating an image to work with. We like the idea of using Zygote because they seem willing to help us develop whatever we need for our project. But, the best order to work would be to consider sensors first, then haptics, then the HMD, and leave the actual image as the last consideration. We need to decide what kind of tracking system we plan on purchasing. Find out how closely they can track accuracy. Find out what kind of support they offer for movement, etc. Start with the sensing system and find out how well they work with the 3D image program we use (either DAZ or Zygote). Ask what HMD’s they recommend and what kind of resolution they know works best with their sensors. Check to make sure of what kind of files they render and then make sure DAZ or Zygote can support those files. Aslo make sure that the haptics we want to use will be compatible. With remote sensing, wireless would be better for the range of movement, etc. Smaller sensors would better fill our needs for this project.

We discussed VRML. It was created to work more quickly with the internet, but the clarity and resolution shouldn’t be affected. We don’t know yet if our expectations are too high for ARToolkit.

It would be a good idea to have students practice the injections with the underlying structures visible, and then practice again with those structures covered to get a better idea of what it will be like on a real person.

Eventually, see-through glasses would be better for this project because then it would not be necessary to attach a camera.

We realized today that not only do we need to be able to receive feedback from the syringe, but also from the other hand as it pulls back the cheek and feels for anatomical landmarks. This will require additional sensors! It will be very difficult to make it feel like you really have your finger in a person’s mouth. The structures won’t manipulate like they would on a real person. For the pilot project we will probably only be able to SEE structures. The most important thing right now is for us to get this pilot project up and running so that we can open the door for future funding to further this research.

When we told Prof. Shelton that we wanted to try and present our research at the international level, he suggested that we look at the ISMAR conference, which is aimed toward this type of technology. He thought it was a great idea for us to present internationally and encouraged us to pursue this goal.

The last thing we did was to take a tour of the Creative Learning Environment (CLE) computer lab, which Prof. Shelton helped to launch at Utah State. We really appreciate all his help today and hope to continue working with him in the future!

2005-08-02T14:32:00.000-07:00

We found out that the meeting for the undergraduate research commitee is to be held sometime this week and therefore our purpose for todays meeting is to prepare our proposal for further funding. We contacted all the companies that we plan to purchase materials from for more specific estimate of cost and detailed information about the products we intend to purchase.

The companies included:

Ascention technologies- flock of birds tracking system
Zygote- male human anatomy 3d image (we discovered this company through a google search which took us to "Turbo Squid" which is an independent contractor company that has compiled a webpage of different 3d images from different companies.)

We emailed the specifics of our project to each of these companies and are now waiting for a response and a price quote.
We have set up a meeting with Brett Shelton, a professor and expert in mixed reality at Utah State University in Logan. He will help us to know how to progress and what the next phases of our pilot project might be.

2005-07-28T11:41:00.000-07:00

Today our objective was to present our research to the Dean of the College of Health Professions. The individuals that were present at the meeting were Shelley Conroy (Dean), Craig Gundy (Assistant Dean), Alan Lewis (Computer Support), Stephanie Bossenburger-james (Director of the Dental Hygiene Program), and Kami Hanson (Dental Hygiene faculty and Project Mentor). We presented eveyone a packet of our research and our future intentions and ambitions for our project. It contained a detailed summary of the pilot project, which we are doing this summer, including conferences we would like to be funded for and a budget of past expenses and proposal for future expenses. The packet included a student request for funding sections which included funding proposals for travel, project supplies, and a stipend for this summers research. It also included a full detailed project description of the projects multiple phases and long-range goals. The meeting commenced at 10:00 am and lasted for about an hour. We described our project and how and why we got started and how we can progress. We also explained that our project full scale can include other departments at Weber State. The 3d image of a skull can include the full body and can be integrated into the nursing program. The 3d image may be able to be produced through the department of visual arts and the specific software programming can be done through the computer science department. We would like the involvment of any department on campus that wants to contribute to the perfection of Weber States own Human Interface Lab. We also consider the contribution of other departments expertise to be a valuable asset to the completion of the project.

Our goal for the meeting was to plant a seed of our ambitions and to help others recognize how valuable and prestigious our project will be to specific health centered programs at Weber State and to the University as a whole.

Future goals

Continue to research different converters and 3d modelers to render a more detailed image
meet with Brett Shelton at Utah State University to collaborate and know how to further progress
Insert the image into the artoolkit coding for the webcam to recognize and then render a 3d image from a pattern
continue our enthusiasm for the research and its benefits to the University and Dental Hygiene Program

2005-07-26T12:12:00.000-07:00

Our assignment over the weekend was to find a converter program that would convert .obj files to vrml. We found a free converter download online called Wings 3D. We import our saved image from Daz Studio into wings 3d and then we export it and save it as a vrml (.wml) file. We then did a google search for vrml plugins which would allow us to view our saved images from Daz Stuio as vrml file. We found several plugins (cosmo player, cortona, and XJ3D) We decided on the cortona. We were able to get it working on both the mac and the pc. Unfortuntely, there is hardly any detail or color to the image. We think it might be a problem with our Daz Studio image export.

Thursday, we have a meeting with the Dean of the College of Health Professions to give her an idea of our research so far and also to give her an idea of where we would like to take our project and what the future goals are.

Future objectives:

How to get better detail on our vrml image
Look into remote sensing equiptment
How do we progress from here?
Meet with and collaborate with team members from Utah State University

2005-07-21T15:20:00.000-07:00

Since we are at a stand still in our progress, we decided to present the research we have done so far to the assistant dean of the College of Health Science at Weber State University, Dr. Craig Gundy. We showed him the 3d image we were able to produce from Daz Studios, we showed him the difference in quality of the graphics between PC's and Mac's. We also showed him the vrml image we are able to render using the artoolkit 2.6 on the PC. We had Dr. Gundy come for a presentation to give us recommendations on how best to proceed with getting more funding for our project. He recommended we give a presentation to the Dean of the College of Health Science, Shelley Conroy. He also recommended that we put in a proposal for funding directly from our specific college instead of from the University.

Today we also took photographs of members of our group using the patterns from the artoolkit and the ar glasses in a clinical setting. We imported these pictures onto a document along with a description of our project and the members involved that will be sent to Utah State University to be posted on their Creative Learning Enviroments website.

We manipulated images in Daz Studio to get the 3d image the way we would be seeing it in a clinical setting. We were also able to turn the muscles on the skeleton 50% translucent so the skeleton could be seen through the muscles, the teeth could be seen through the cheek and jaw muscles. This is important for the whole purpose of our project, to help students to visualize the landmarks needed to give local anesthetic.

Alan worked on our project on his own computers (mac 10.4 and mac 10.3). He was able to get artoolkit rendering images. He put the programs he used, and a copy of his own computers operating system onto a portable hard drive. He concluded that their is something in common with the laptop mac computers that our group is using the is disrupting the functions of the artoolkit. He wants to try finding what that problem is but the alternative or "quick fix" would be to just use the USB ports on our laptops and download the programs that he has on his portable hard drive.

We hope to have our funding in a couple of weeks and be able to progress our project further.

2005-07-19T08:29:00.000-07:00

As we are in the beginning phases of this four year project, we have the task of troubleshooting all the problems that occur. Our progression is somewhat stalled because we are in need of more funding for haptics (from SensAble technologies) and other essential materials. We are also stalled on our progress with the mac computer because we keep getting warnings and errors when we try to install and build all the programs we need to use the newest version of artoolkit (2.70.1). We went to the README file for instructions on how to build artoolkit using a mac and we got to the process of installing Mozilla-dev when on one mac computer a warning message appeared:

Warning: if you compile x11 applications against this xfree86 release, you will not be able to run them if you decide to revert to apples x11 provided with panther. You have an existing x11 installation in /usr/x11r6 and or /etc/x11. This package refuses to overwrite these. Remove them then tell fink to install xfree86 again. The package won't recompile. If you want to keep your x11 installation, install system-xfree86 resp. system-xtools instead to make this known to finks package system.

On another computer we got a different error:

Failed: no package found for openvrml4-dev and openvrml5-dev

For both of these errors we have found no solutions. We copied and pasted the errors into a google search and came up with the artoolkit mailing list archive. Other users of artoolkit on the mac have had the same problems and the solution given was to update fink commander by going to source, update-cvs in the fink commander utilities bar. We tried this on all computers and when the selfupdate was complete it recommended that fink update all of its packages. So then we ran source, sef-update, update-all. After this was completed we then again tried to follow the instructions on the README file but the same errors kept appearing.

We continued to troubleshoot by uninstalling and then reinstalling the programs need for this project (x11, xcode, artoolkit, fink) But with no luck.

The artoolkit( old version) works well with the PC and the ar glasses work but the only obstacle is converting our .obj image we saved from daz studio into a vrml image. For this we need a converter but our attempts to find a converter online has been slow. We found one called uvmapper but with this we ourselves have to reconvert the blob the appears back into our image. We will continue to research converters. So for now our goal is to find a converter and produce our 3d image on the PC using the artoolkit patterns. We will have to retrain the existing pattens to recognize the specific image we have saved.

Alan gave us some good recommendations to increase our knowledge base of computers, esp. mac. He recommended we purchase the "missing manual" series which is a users guide to mac computers. He also told us about another department at Weber State University, the physics department, doing a large project using macs. The head of the department is John Armstrong and the project is called SAVI. We may look into collaborating and sharing information.

Our project is in collaboration with the instructional technology department at Utah State University. They have invited our Augmented Reality group to present our research at the IT conference this fall. We are also submitting proposals for funding to attend the siggraph conference in Los Angeles 2005 and two other international conferences in Italy and Austrailia. We are hoping that we can establish an international connection for Weber State.

Our future goals are:

Find a converter
retrain an artoolkit pattern to recognize our image
get more funding from undergraduate research
eventually get the artoolkit to work on the mac

2005-07-14T14:42:00.000-07:00

We petitioned the help of a Weber State University employee named Alan Lewis who is an expert in Mac computers to help us with our augmented reality project. The objectives for today were:

How to render images using the mac artoolkit

We compiled xtools with the new version of artoolkit(2.70.1) found at sourceforge.net/projects/artoolkit, using the terminal and the instructions off the readme file. We then downloaded mac X11 and finkcommander at apple.com. These programs are all required to render 3d images in artoolkit. We then were able to render a 3d image from the sinplelite file of a multi-colored cube using the Hiro pattern. The other patterns and files were not accessible as yet.

Alan was then gracious enough to give us a general tutorial session on how to use the terminal and the meaning behind all the symbols and file abbreviation and other keyboard commands. He also explained basic mac computer functions. He is a valuable resource to our project and to the University and we hope that he continues to share his knowledge base with us.

Today we also referenced a souce of information from Lynden Jones, who is a computer programmer at IHC. He was able to explain over the phone part of the compilling process and help us to render the image in simplelite. He is also a very valuable resource that we will hopefully be able to use in the future.

Future objectives:

Vrml image rendering

Inserting saved images into artoolkit coding

Folllow up on questions noted on July 7,2005

2005-07-12T14:52:00.000-07:00

Today our objectives were as follows:

Investigate web browser plugins.

We went online and searched for Vrml browser plugins to access onine 3d images. We downloaded the XJ3d browser from mac3d.org that is compatible for mac computers and mac 3d images.

Research websites to build our knowledge base on augmented reality.

We visited and bookmarked the websites for the Washington Hitlab, Dart (gvu.gatech.edu/dart/), MIT(csail.mit.edu) and macweb3d.org.

Detail literature review for bachelors thesis paper.

The outline is in printed form and each member of our group has a copy it was also sent electronically via email. This outline will help us to get started in August 2005 on our bachelors paper for our bachelors degree.

Interface daz studio graphic with artoolkit.

We are not able to accomplish this unitl all mac computers have downloaded xcode-tools, the equivalent of microsoft visual studios, to be used with the mac version of artoolkit. We will continue to work on this objective.

Mac webcams.

We purchased mac compatible webcams and installed them on each computer.

Future objectives will be:

Start a search for haptics rendering software/programs.

We have been researching a company called "Teleos." Although our research has been extensive we have been unable to locate a website or any form of contact to purchase their haptics software. We found a name and phone number supposably from teleos research but only got an answering machine when we called (Ann Reid 415-328-8800) We will continue to look for other companies as well as continue our search for teleos.

2005-07-07T13:40:00.000-07:00

We continued to work on the Daz Studio 3d images. We purchased muscle maps so now our image can be skeleton, skin, or muscles. The anatomy on the 3d figures is very realistic and detailed which serendipitously can be used also in the Dental Anatomy class required for Dental Hygiene. The steps for creating an image are as follows:

select MS or V3 nonconforming skeletons

then choose M3SRI under Daz Studio People

highlight M3SRI and "fit to" skeleton (fit to is located under parameters)

go into "scene" choose M3SRI, poses, Daz skeleton, and the choose the zero figure

go into "scene" choose M3 skeleton, poses, Daz skeleton, and choose the zero figure

Now you can select which translucency you prefer but make sure M3SRI is highlighted.

now you are able to inject all or some expressions/morphs. (INJ = injected, REM = remove)

Questions that we had while we were manipulating our images:

Why does the image have two sets of teeth when the mouth is opened?
the answer is because the skeleton and the muscle images are overlapped onto each other and both have a set of teeth. You need to turn the visiblity off on one set.

When we save the 3d image that we have created, how do we get the object to respond to manipulation? Is this part of haptic authoring or do we need to create other 3d images?

How do we render the image and its responses all at once in a smooth sequence?

How do we render images in our HMD glasses as an image only and not including the computer screen?

Do we need to attach our web camera to our HMD?

Is the HMD Mac compatible, as we are all using mac notebook computers?

As we continue our research we hope to find the answers to these questions and will probably reveal more questions that will help our AR project to progress.

2005-06-30T14:55:00.000-07:00

As we met today we mostly experimented with the DAZ Studios that we downloaded on Tuesday. We downloaded the base characters V3 and M3 and familiarized ourselves with the program contents. Part of this familiarization included individual experimentation and collaboration as well as working through the tutorial.

We recently received an email invitation to participate in the International Conference on Entertainment Computing concerning Advanced interaction design i.e. haptics; education and training technologies; augmented,virtual and mixed realities. This conference will be held in Sanda, Japan at Kwansei Gakuin University on September 19-21. Today we began a proposal for funding to go to this conference using LaTex format.

One question we had today was about Microsoft visual studios and its significance in Artoolkit. Their is a downloadable version of Artoolkit for a MAC but Microsoft visual studios is not compatible on a MAC. We also have downloaded DAZ Studios which is similar to Microsoft visual studios but is compatible with a MAC. Why is Micorsoft visual studios needed if not compatible on all computers and do we need two image rendering programs?

2005-06-28T14:01:00.000-07:00

The last two weeks have been spent at ADHA( in Las Vegas, NV) and other science related workshops and meetings(in Hawaii). As we met back with each other after two weeks we realized that we needed to update our Artoolkit with the updated version on the washington Hitlab website.
We also were able to render images using the vrml artoolkit that we were unable to do so before. During the two weeks we did not meet we each individually researched 3D cranial models, which will be coded into the artoolkit so that when an image is rendered we will see the cranium. Today we reviewed those possibilities and purchased a package of virtual muscle and skeleton images of the cranium and the entire body we also had to download Daz Studios, spot rendering and the base from Dazproductions.com or Daz3d.com so that we were able to view the images. After we downloaded all we thought necessary, some of our computers signaled a "crash" impending on the computer. We discovered that we need a high powered graphics card. One that is compatible with openGL with 128 MB of ram. The one that daz recommended was Nvidia geforce series.
We also contacted Teleos research about haptics for our virtual image. We were unable to contact a person but left a detailed message on the answering machine. The Teleos website is very difficult to find and once found, access is not granted unless you have a correct user name and password. So we contacted a phone number found on a website we found doing a google search of "Teleos" and will hopefully be responded to.

2005-06-09T10:15:00.000-07:00

Today we started familiarizing our group with computer software programing and microsoft visual studios. We purchased our Head mounted display (i-glasses) and we started to use the image cards we printed from the washington hitlab website to render images with our webcam. We reviewed the archive of the artoolkit mailing list to see if any other groups or persons had asked the same questions we had. Our questions were

How do we render VRML (virtual reality modeling language) images?
What is the best HMD to use?

We looked at i-glasses which has a resolution of 640x480 which is pretty poor but usable

We also looked at the olympus(eyetrek)FMD 700 but it did not have see through mode

We looked at cy-visor at wwwpersonaldisplay.com but those did not have see through mode either

We researched glasstron by sony but those were out of our price range

What does msvcirtd.dll mean and if it is a file how do we find it?

At first we thought it was part of a fire wall but we ran an antispyware program and it still gave us the msvcirtd.dll missing file error

msvcirtd.dll is a file library

msvcirtd.dll and msvcrtd.dll are both available in artoolkit 2.65 (not vrml) and we copied and pasted those files from 2.65 to artoolkit direct show 2.52 vrml and the missing file alert did not come up and we were able to render and image on our home computers

We are not computer programers but we need to know some things about coding in order to get the image that we need. We are and will be continuing to get familiar with coding by doing the examples on the artoolkit manual.

2005-06-07T10:02:00.000-07:00

We started working on our Augmented reality project for Weber State Dental Hygiene students. Our purpose is to use augmented reality to teach and learn local anesthetic. We started off by reviewing our calender for the summer and then researching and downloading those programs and documents needed for our project.

We downloaded Artoolkit on our home computers
We practiced rendering images using artoolkit and a webcam
We researched the best head mounted display available in our price range
We purchased personal webcams for home instant message conferences
We subscibed to the artoolkit mailing list to explore others questions on augmented reality
We read and followed the tutorial on artoolkit